Crusty Blaa

The OpenStack Foundation Board of Directors met for a two hour conference call yesterday. The usual disclaimer applies - this my informal recollection of the meeting. It’s not an official record. Jonathan Bryce has posted an official summary of the meeting.

Executive Director Update - 2017 Budget

After taking a roll call and approving minutes from two previous meetings, Jonathan gave a presentation on the 2017 Foundation budget.

Jonathan described a number of goals under a "One Platform" theme for 2017. He talked about how OpenStack should (a) make sure containers are first class citizens within OpenStack, (b) ensure workload portability across interoperable public and private OpenStack clouds, and (c) reinforce OpenStack position as the standard platform for NFV.

Jonathan went on to remind the board that in 2016 we had planned for a 2016 loss, the idea being to invest some of the cash reserves that the Foundation had built up in order that we could take on some new activities. In 2017, the proposal is to aim for a break even budget so as to avoid not further depleting the reserves. However, with the addition of new Gold Members, we will be able to maintain the increased investment from 2016 while still breaking even. Mark Collier noted that China is a key growth area, given that much of this new funding is from new Gold Members from China.

Jonathan then presented the budget itself, showing quarter-by-quarter income and expenses in a number of high level buckets like "Corporate Fees", "Events", "G&A", "Community Development", and "Community Infrastructure". One particularly striking aspect of OpenStack Foundation budgets is that much of the financial activity comes in Q2 and Q4 with the OpenStack Summit events in those quarters, accounting for $10.5M and $6.5M activity in Q2 and Q3 respectively out of the total $22M budget.

A discussion followed the presentation around what level of financial detail the board would like and expect to have available. Roland Chan expressed a concern that he was unable to tell how the strategic decisions the board had made were impacting the budget. For example, you could compare the 2016 and 2017 high-level budgets and make a good guess as to the impact caused by the addition of the PTG, but it would be nice to understand that in more detail. Some debate ensued as to whether this was information that was already available to the Finance Committee, whether the board should be supplied with a more detailed budget breakdown, or whether we instead need some analysis and commentary about the financial impact of key changes.

Jonathan took an action to prepare some further information which should help address the question. Alan also took an action for the Finance Committee to consider how to address this more structurally. Finally, the 2017 budget was approved by a vote of board members.

User Committee Proposal

Next up, Edgar Magana presented some proposed by-laws changes around the structure of the User Committee and related changes to the UC charter.

Since these changes had been previously discussed by the board, they were put to a vote after some short discussion. The board approved a motion to replace section 4.14 of the by-laws with the text above, along with adding a new UC member policy appendix.

OpenStack Futures

The board then turned its attention to a discussion started at the previous board meeting about the future of the project and some key questions it was felt the board should be considering.

Allison Randal lead the discussion by talking through some of the information the board had gathered in an etherpad since the previous board meeting. We had condensed the questions and concerns into four distinct areas:

1. OpenStack and its evolving relationship with adjacent technologies.
2. Unanswered requirements, and how they will be address by the project over time.
3. Whether OpenStack is sufficiently able to evolve architecturally architecture to meet certain needs.
4. Community health indicators the board and the wider community should be paying attention to.

It was felt that we would only be able to discuss these in a useful way with an in-person board meeting, and the hope is that we will be able to hold this meeting alongside the PTG with involvement from the Technical Committee.

In order to help frame the discussion at that in-person meeting, Allison proposed that board members consider a series of 12 strategic questions as homework for the next board meeting. These questions are based on the 12 "boundary questions" proposed by Werner Ulrich in his work on Critical Systems Heuristics (CSH). There was some discussion as to whether these questions could, in future, be posed to the wider community via a survey.

There was some brief discussion about how this relates to "OpenStack-wide Goals" mechanism recently introduced by the TC. Toby Ford gave the concrete example of "1000s of nodes under the management of a single control plane", and wondered whether that would qualify as an OpenStack-wide goal. Doug Hellmann happened to be on the call and was able to explain that the goals mechanism is used to achieve changes that address concerns across all projects and which can be completed in a single cycle.

The discussion ended with the board agreeing to complete the 12 questions prepared by Allison for an in-person meeting in Atlanta.

Gold Membership Expansion

The final topic of discussion was raised by Shane Wang. Now that the 24 Gold Member slots have been filled, an obvious question remains ... should the Foundation increase the number of Gold Member slots?

Shane made the case that more companies want to join as Gold Members, particularly in China and gave the example of an other major Chinese telecommunications operator that is eager to join.

Roland explained that the topic had been discussed at a recent - but poorly attended - meeting of the Gold Members. It was felt at that meeting that more options need to be considered than simply increasing the number of Gold Member slots - for example, increasing the number of Platinum Member slots so that perhaps some Gold Members could move to Platinum, or the introduction of a Silver Member tier.

However, it was also felt that we should carefully consider our goals even before jumping into considering specific options. Anni made the point that we're trying to achieve a balance between the risk of devaluing the membership status versus wanting to welcome more participation in the Foundation.

All felt that this was an important topic that warranted further discussion and consideration.

Looking Ahead to 2017

And so, the boards work for 2016 has come to an end. The schedule of meetings for 2017 has not yet been finalized, but the next two meetings are likely to be a two hour conference call on January 31 and an all day in-person meeting in Atlanta on February 24.

This is the prose version of a talk I gave today at OpenStack Day, France. The slides are available here.

Today, I want to speak about a somewhat subtle, meta-topic. I hope to share some insight on the question of how a company should think about its investment in any given open-source project. The reason I think the subject is important right now is that, as OpenStack follows its inevitable path through the “hype curve”, we are seeing some major players in the OpenStack ecosystem have recently re-shaped their investment in the project. I think there is plenty of opportunity for companies to take a much more realistic view on this subject, and my hope is that this will lead to a much more sustainable level of investment in the project.

My Story

My views on this question are so heavily influenced by an early personal experience, that I’m tempted to talk at length about my own story. But time is short. Very quickly, though, as I was finishing university, I realized I wanted to focus my career on open-source, and I was faced with the obvious question of who was going to pay me to work on open-source? What value would be open-source work bring to my employer? I spent a lot of time thinking about this, even after I found myself employed to work full time on open-source.

My Employer

An interesting thing happened. Because I had spent a good deal of time thinking about the value of my own open-source work, I was well-placed to help my employer think about how, where, and why to invest in open-source projects. It quickly became apparent to me then - and is still apparent to me now - that this fundamental question is fraught with difficulty, and most people struggle greatly to answer it.

Investment vs Business Needs

Over the years, I’ve boiled my answer to this simple-sounding question to something equally simple - your investment in a project should be directly related to what the business needs from the project. I think it’s important to frame it this way, because if the business doesn’t have a good have a good understanding in the value of an investment, the business is going to be quick to discontinue that investment when it is looking at how to make the best use of its resources.

Anti-Patterns

What do I mean about an investment that isn’t directly related to the needs of the business? Let me dig into that a little bit with some anti-patterns.

Focus on Features

First, features. Or, as Thierry Carrez puts it, “tactical contributions”. Companies often look at the “what do we need from this project” question through the lens of feature analysis - what are my requirements? What is missing?

The good thing about contributing features is that it is directly related to business needs. The thing about your primary focus being on new feature development is it misses the bigger picture. A community of developers that is focused only on their own new features is not going to get anywhere. Who is thinking about the long-term future of the project? Who is looking at feedback from users? Who is merging the code for these new features?

All of these types of activities are the basic necessities of any software project. They are the basis through which new features get added. You must invest in ensuring that this is happening in this project whose future you care about.

The Donation

This anti-pattern is about our choice of language, and how it affects our thinking. People often talk about “donating” code to a project. Calling it a donation suggests you have a feeling that the return on your investment is quite intangible. How long will you continue with these “donations”?

For Recognition

Related, it’s often quite clear that a major motivation for companies investing in open-source is the recognition they receive in doing so. Certainly, in OpenStack, with Stackalytics we have spawned an impressively pointless competition between companies for ranking in contributor statistics. If you give your employees the goal of achieving a particular ranking in the contributor statistics, you may achieve this, but what does that really achieve?

The type of business value that it delivers is essentially short-term marketing value. We don’t want investment in open-source projects to be made out marketing budgets, since that’s possibly the least reliable source of funding!

"100% Upstream"

Not so long ago, when companies were falling over themselves to demonstrate their commitment to OpenStack, we saw an interesting phenomenon - the “100% dedicated upstream resource”. These were individuals and teams at various companies who were employed to contribute to the project but - as far as I can tell - were self-directed and deliberately kept isolated from any “downstream” work.

This is an incredibly alluring opportunity for those involved! The company is saying that the project is critically important to the business and whatever you or your team does to help the project be successful, that’s valuable to the company! Unfortunately, we can see how these things go in cycles and, at some point, the company will take a harder look at what this person or team is doing. When that happens, the likelihood is that the company has very little visibility into - or understanding of - the work being done and, worse, the person or team has no experience articulating how the value of the work is meaningful to the business!

There are some exceptions to this, even within Red Hat. But as a systematic way of investing in a project … it’s unlikely to be sustainable.

Non-Profit Staff

Finally, the culmination of a number of these anti-patterns - the idea that companies should "donate" to a non-profit organization like the OpenStack Foundation, have that organization use the donations to employ staff who are "100% dedicated upstream resources", and the value to companies ...? The recognition it receives for being such generous benefactors?

Having some technical staff employed by the non-profit is fine. Some of my favorite people work for the OpenStack Foundation! My preference would be for Foundation staff to be in facilitation roles. But certainly, you would do well to avoid this pattern for the majority of the contributors on a project.

One example of this is the Linux Foundation's Core Infrastructure Initiative. In the wake of the OpenSSL Heartbleed vulnerability, the Linux Foundation brought together funding from a number of companies to invest resources for key projects like OpenSSL. It's fascinating because here is a project like OpenSSL that many, many businesses depended on, but few invested in. What the Linux Foundation has done is positive, but I can't help feel we have failed as a community if this is the only way to sustain these projects. You'll notice that Red Hat hasn't joined this initiative, despite us being supportive of it - we have always invested in these projects directly, and think that is actually a healthier model for the future.

Think Strategically

Ok, so that's a bunch of anti-patterns ... what you generally shouldn't do. What should you do? Well, you need to think strategically. You are choosing to have your business depend on something you don't control, but you should not allow yourself to feel powerless about how successful that choice will be.

The Future

Think about the future, the long-term. Given the business choice you are making, for how long will your business success depend on the success of the project? Indefinitely? What are you doing to ensure that success?

One way to think about that would be a worst-case scenario. The unimaginable happens, and the rest of our community disappears. You are left almost completely alone maintaining the project. What you focus on? Obviously, you wouldn't choose to depend on a project where there is a possibility of that happening, but the thought exercise does help you think about your priorities.

Influence

If you are determined to ensure the success of the project, you'll have your own view of what that success looks like. Are you happy to hope the community will find a direction that fits your needs, without any input from community leaders that understand the needs of your business?

At Red Hat, we talk about wearing "two hats". Given a particular problem, a particular question of direction, you should have the ability to understand what's good for the project overall and what's good for Red Hat. And crucially, you should have a way to reconcile those two views. This is not a zero sum game. Almost always, you can find a solution that is good for both the project and Red Hat. Why would Red Hat want a solution that is harmful to the project?

Expertise

In order to be successful with any technology, you need to have access to expertise that understands the technology. There is no better expertise than the authors or maintainers of the project. These are the people who understand not just how the technology works now, but how it evolved there, and how things are likely to change in the future. They understand the pitfalls, the know the cool new tricks.

You can have access to that expertise by having those experts join your team. They stop being experts pretty quickly if they stop having time to work on the project, so their upstream responsibilities should continue to be a significant proportion of their time. But you'd be amazed at how their presence on the team can help the whole team be successful.

Red Hat's Model

As somewhat of an aside, since this presentation is not a sales pitch, think about Red Hat's business model, and our proposition to our customers. Certainly part of the model is that, through our product subscriptions, the customer is investing in the projects involved and, by proxy, is safeguarding the future of the project, gaining a level of influence in the project, and has access to expertise relating to the project.

A Measurable Goal

Recently, on Red Hat's OpenStack team, and thanks to the influence of Alexis Monville, we've been using the "Objectives and Key Results" framework for using well-defined, measurable goals to guide our teams. We started brainstorming on these OKRs almost a year ago, and from those early discussions, I wondered “how do we frame an measurable goal around our investment in upstream?”. What indicator could we use to know whether we were on the right track, and to ensure we’d stay on the right track?

Our Vision

Our first thoughts on this was to look at data like our position in the contributor statistics, or the number of core contributors, PTLs, or TC members on our team. None of this sat well with us, though, because we have seen that these type of goals don’t drive the right behaviors. When we started drafting a vision statement for each goal, I wrote:

“Teams of engineers with the required motivation, an understanding of Red Hat's vision, and empowered with the necessary time, encouragement, and recognition, working upstream to drive and influence many diverse parts of OpenStack.”

And there it sat for a while, us all lacking ideas on how to measure this. Quite recently, Russell Bryant and Doug Hellmann hit on a promising solution. Survey the team with a small set of questions and use that sentiment of our measure of success with this goal.

The Questions

The questions that Russell and Doug developed are:

1. Does your team have effective input into the features accepted and design decisions made upstream?
2. Does your team have effective input into bug fixes and backports made upstream?
3. Does your team have effective input into discussions related to processes, schedules, requirements, infrastructure management, and other decisions made by the upstream OpenStack community in general?
4. What level of investment does your team make in upstream work?

The allowed answers are “not enough”, “just right”, “too much”, and “don’t know”. We also had a free-form comments section which is helping us gain insight into the results.

Notice one important aspect of this - we are asking individuals about the effectiveness of the investment their team is making upstream. That reflects the vision above.

The Results

We only recently started running this survey, and we will do one every release cycle. So far, we’ve had over 70 responses, so that’s a pretty good start.

The really interesting judgment call we need to make is what percentage of “just right” answers do we want to aim for? It would seem misguided to aim for 100% - do we really think that it’s important that every individual feels we’re striking the right balance? We’ve arbitrarily chosen 80% as our target, which means we feel pretty good about where we’re at, but there’s still opportunities for improvement.

Sustainability

One thing I’m loving about my job these days is the direct exposure I have to Red Hat customers who are choosing to invest in OpenStack. Deciding to build a significant cloud for production use is a huge decision, and no-one takes it lightly. It’s exciting, and companies doing this can look forward to a true transformation in how they think about IT resources. The truly daunting part is the sense of responsibility that comes with it.

These are long-term choices being made, and they’re being made because people believe in the future of this project. Red Hat takes the responsibility seriously by making choices that we hope will ensure the project has a long, sustainable future.

I had previously posted summaries of most board meetings, but had stopped doing this when I was appointed as Red Hat's representative on the board. Lately, I've sensed folks at Red Hat becoming more interested in the workings of the Foundation, so I figured it might be useful to start doing this again.

The OpenStack Foundation Board of Directors met for a two hour conference call last week. The usual disclaimer applies - this my informal recollection of the meeting. It’s not an official record.

Gold Member Applications

The first half of the meeting was given over to an Executive Session, after which the board voted to approve China Telecom, Inspur, and ZTE as Gold Members of the Foundation.

This completes the handling of the 7 Gold Member applications that were presented to the board at the October 24 meeting in Barcelona. 99Cloud, China Mobile, City Networks, and Deutsche Telecom were approved at that meeting.

For the first time, the Foundation now has reached the maximum of 24 Gold Members. We will only be able to consider applications for new Gold Members if an existing member departs, or if we go through a difficult process to change the limit in our bylaws.

User Committee Proposal

Next up, Edgar Magana updated the board on some proposed changes to the User Committee that was first discussed at the October meeting.

Currently the bylaws describe the User Committee is an "advisory committee" of three members appointed by the Board and TC, which prepares regular reports for the Board and TC. The idea with this proposal is to make the User Committee a parallel entity to the TC, with its members chosen through an election of Active User Contributors (AUC).

The bylaws changes outline how the User Committee has at least five members, that elections of Active User Contributors (AUCs) are held every six months, an affiliation limit for members of the UC, how AUC status is determined, and more.

The hope is that feedback will be collected as comments in the proposal document and that the Board will vote on the proposal during our December 6 meeting. A vote of the Board is sufficient to enact the changes.

One point of discussion was whether bylaws changes are necessary at all. The hope when the bylaws were originally drafted was that the User Committee would have plenty of lattitude to evolve via changes to the UC charter. Edgar made the point that the key change is for the UC to become a parallel entity to the TC rather than an advisory committee to the Board and TC.

"Futures" Discussion

Next, Toby Ford gave a presentation on a topic he had proposed along with Imad Sousou, Allison Randal, and Mark Baker. Toby introduced the topic by saying that as any project evolves and matures, it is important to reflect on competitive threats and the need to evolve the project.

Toby talked about a wide set of competitive threats to OpenStack from the container ecosystem (including Kubernetes and Mesos), to other storage implementations, to various projects in the Telco world (OpenVIM, E2, 3GPPP), and the obvious challenge of AWS, Azure, and Google with AWS revenue over $10B.

Toby moved on to talk about "the good and the bad of the Big Tent", describing the need for a balance between diversity and innovation versus consolidation and refactoring. Toby made the case that we're seeing a lot of expansion in the Big Tent, but not consolidation and refactoring. He expressed particular concern about how and whether core projects will be able to evolve, especially if the Big Tent does not allow for competitors to existing projects.

Toby then put on his "AT&T hat" and talked about the challenges for OpenStack from an AT&T perspective - the need to get to a control plane that scales to 10k servers, the problem of managing over 100 different production sites, the challenge of more and more happening "at the edge" with 5G, innovation happening in the networking space and how it relates to OpenStack, and the unsolved problem of keeping a production environment current with latest OpenStack releases.

To wrap up his presentation, Toby listed a few "possible recommendations" the Board could make to the technical community - (1) allow core projects to have competition within the Big Tent, (2) a mechanism to incubate new approaches, and (3) a mechanism to reationalize, clean up, or refactor.

What followed was a pretty lively discussion that covered much ground, over and beyond the concerns raised by Toby. While the success of OpenStack in bringing together such a diverse set of interests was acknowledged, there was a definite frustration that some form of necessary change is failing to emerge naturally, and whether it falls on the Board to try to at least try to clearly articulate the problem at a strategic level.

Jonathan tried to focus the conversation by giving his perspective that the Big Tent concerns had tied down until "recent comments in the media", and that is probably a distraction from the concerns people are expressing about core projects like Nova and Neutron. He was at pains to say that this isn't about project teams doing bad work - we continue to make huge progress in each release, and valuable work is being done. However, we're definitely seeing frustration from some quarters that it is difficult to influence the community with their ideas.

Jonathan warned that talking too much in the abstract creates the risk that the discussion will go around in circles. Despite this, the discussion never did go into any particular detail on where we've witnessed the problems we were discussing. From my own perspective, it was clear that much of frustration was as a result of how the Ciao and Gluon projects have been received, but we're failing to learn anything significant from those experiences by not talking about them in detail.

By the end of the discussion, we had agree to collaborate on a concrete description of specific problem areas, the questions they raise, and some possible solutions. The hope is that we would complete this before our December meeting, and we may then plan a longer meeting (possibly face-to-face, possibly with the TC) to dig into this.

Derek mentioned "upstream packaging" on this week's packaging meeting and asked RDO packagers to participate in the upstream discussions. I thought some more context might be useful.

First, a little history ...

When I first started contributing to OpenStack, it briefly looked like I would need to make some Ubuntu packaging updates in order to get a Nova patch landed. At the Essex design summit a few weeks later, I raged at Monty Taylor how ridiculous it would be to require a Fedora packager to fix Ubuntu packaging in order to contribute a patch. I was making the point that upstream projects should leave packaging to the downstream packaging maintainers. Upstream CI quickly moved away from using packages after that summit, and I've heard Monty cite that conversation several times as why upstream should not get into packaging.

Meanwhile, Dan Prince was running the Smokestack CI system at the time, which effectively was being treated as OpenStack's first "third party CI". Interestingly, Smokestack was using packages to do its deployment, and for a long time Dan was successfully keeping packaging up to date such that Smokestack could build packages for patches proposed in gerrit.

And then there's been the persistent interest in "chasing trunk". Operators who want to practice Continuous Deployment of OpenStack from trunk. How does packaging fit in that world? Well, the DevOps mantra of doing development and CI in environments that model your production environment applies. You should be using packaging as early on in your pipeline as possible.

My conclusion from all of that is:

1. A key part in building a Continuous Delivery pipeline for OpenStack is to practice continuous package maintenance. You can glibly say this is "applying a DevOps mindset to package maintenance".
2. How awesome would it be if OpenStack had "upstream infrastructure for downstream package maintainers". In other words, if downstream package maintainer teams could do their work close to the upstream project, using upstream infrastructure, without disrupting upstream development.

I think the work that Derek, Alan, Dan, John, and everyone else has been doing on Delorean is really helping RDO maintainers figure out how to practice (1). I first started maintaining Fedora packages for Fedora Core 2, so IMO what RDO is doing here is really dramatic. It's a very different way of thinking about package maintenance.

As for (2), this where we get back on topic ...

At a Design Summit session in Vancouver, the idea of maintaining packaging using upstream infra really took hold. Thomas Goirand (aka zigo) proposed the creation of a "distribution packaging" team and this triggered a healthy debate on openstack-dev. Derek has since pushed a WIP patch showing how RDO packaging could be imported.

There's a clear desire on the part of the Debian and Ubuntu package maintainers to collaborate on shared packaging, and it sounds like this goal of further collaboration is one of the primary motivators for moving their packaging upstream. This makes a lot of sense, given the shared heritage of Debian and Ubuntu.

The RDO team is enthusiastic about adopting this sort of upstream workflow, but the Debian/Ubuntu collaboration has added an entirely new aspect to the conversation. Despite the fact that RDO and SUSE platforms have little in the way of shared heritage, shouldn't the RDO and SUSE packaging teams also collaborate, since they both use the RPM format? And perhaps deb and rpm maintainers should also collaborate to ensure consistency?

To my mind, the goal here should be to encourage downstream packaging teams to work closer to the upstream project, and have downstream packaging teams collaborate more with upstream developers. This is about upstream infrastructure for downstream teams, rather than a way to force collaboration between downstream teams, simply because forced collaboration rarely works.

For me, what's hugely exciting about all of this is the future prospect of the package maintainers for different platforms adopting a "continuous packaging" workflow and working closely with project developers, to the extent that packaging changes could even be coordinated with code changes. With its amazing infrastructure, OpenStack has broken new ground for how open-source projects can operate. This could be yet another breakthrough, this time demonstrating how a project's infrastructure can be used to enable an entirely new level of collaboration between package maintainers and project developers.

This week's launch of OPNFV is a good opportunity to think about a simmering debate in the OpenStack developer community for a while now - what exactly does NFV have to do with OpenStack, and is it a good thing?

My own “journey” on this started exactly one year ago today when I visited a local Red Hat partner to talk about OpenStack and, towards the end of our Q&A, I was asked something like “will OpenStack support NFV?”. I’d never heard of the term and, when the general idea was explained, I gave a less than coherent version of “OpenStack implements an elastic cloud for cattle; this sounds like pets. Sorry”. After the meeting, the person who asked the question forwarded me an NFV whitepaper from October 2012 and, glancing through it, most of it went right over my head and I didn’t see what it had to do with OpenStack.

Since then, Chris Wright has been patiently talking me through this space and gently trying to get me over my initial skepticism. Chris would say that our conversations has helped him refine how he explains the concepts to open-source developers, and I think he really nailed it in his keynote at the Linux Foundation’s Collaboration summit in April.

[embed]https://www.youtube.com/watch?v=SAimeBttapA[/embed]

In his keynote, Chris talks about the benefits of collaboration in open-source and walks through all of the various aspects of how the networking industry is changing, and how open-source is playing a key part in all of those changes. He covers, and simplifies:

• Taking the current architecture of proprietary, expensive, complex, difficult to manage forwarding devices (like routers) and how SDN (Software Defined Networking) aims to “put an API on it”. This is what’s meant by “disaggregation of the control plane and data plane” - that forwarding devices become devices which are controlled by open standards, and allows your distributed system of forwarding devices to be controlled and automated.
• NFV (Network Function Virtualization) as a shift in the telco data-center world which embraces many of the lessons that the elastic infrastructure cloud has taught the IT industry. More on that below.
• Changes in the “data plane” world, where we’re starting to see the network device market mimic the x86 server market such that these devices can be “white box” servers running open-source software. Again that disaggregation word, but this time it’s about “disaggregation of hardware and software” and how the software part can be open-source implementations of optimized packet-forwarding capabilities which we’re used to seeing implemented in expensive and proprietary hardware appliances.

But let’s focus here on NFV.

I real don’t know much about the telco industry, but what Chris has me imagining now is data-centers full of proprietary, black-box hardware appliances which are collectively know as “network functions” or “middle boxes”. These boxes are used for everything from firewalls, NAT, deep packet inspect (DPI), the mobile packet core, etc. These are software applications trapped in hardware. They’re expensive, proprietary, slow to roll-out, don’t always scale well and are hindering telco service providers as they attempt to react to a rapidly changing market.

NFV is about completely re-thinking the architecture of these data-centers. This is the telco industry re-imaging their data centers as elastic infrastructure clouds running their “network functions” as virtualized, horizontally scalable applications on these clouds. The exciting - simply stunning - aspect of all of this for me as an open-source advocate, is that the telco industry is settling on a consensus around an architecture involving open-source generally and OpenStack specifically.

Say that again? These huge telcos want to rebuild their entire data centers with OpenStack and open-source? Yes.

If, like me, you want to see open-source change the IT world into one where we all embrace the opportunity to collaborate in the open, while still successfully building building businesses that serve our users’ needs … then this sounds pretty cool, right?

If, like me, you want to see OpenStack as the standard platform from which many of the worlds’ elastic infrastructure clouds are built ... then this sounds like a no-brainer, right?

Well, the thing we need to bear in mind is that these applications (i.e. network functions) are pretty darn specialized. They need to have a high level of performance, determinism and reliability. But that does not necessarily mean they are “pets” and missing one of the key points of an elastic cloud.

Let’s take the reliability requirement - when these network functions are implemented as horizontal scale-out applications, they will look to achieve high levels of reliability in the same way that typical cloud applications do - with each tier of the application spread across multiple failure domains, and by spreading application load horizontally. Telcos will just want to take this further, with faster and more deterministic response to failures, while also avoiding any compromise to application's performance. For example, you’ll see a lot of interest in how instances are scheduled to take to into account affinity and anti-affinity within an instance group.

The performance requirement is largely about high-performance packet processing. How to get a packet off the network, into a VM, processed quickly and back out again on the network. One of the techniques being pursued is to give VMs direct physical access to the network via SR-IOV which, in turn, means the compute scheduler needs to know which physical networks the NICs on each compute node has access to.

The deterministic requirement is about predictable performance. How to avoid the vagaries of the hypervisor and host OS scheduler affecting these performance-sensitive applications? You’ll see work around allowing operators to define flavors, and application owners to define image properties, which between them control things like vCPU topology, vCPU to pCPU pinning, the placement of applications in relation to NUMA nodes and making huge pages available to the applications. Compare to Amazon’s memory-optimized and compute-optimized flavors, and imagine this being taken a step further.

Oh, and another requirement you’ll see come up in this space a lot is … IPv6 everywhere! I’m certainly down with that.

Want to learn more about the work involved? See the OpenStack NFV team's amazing wiki page which goes into excruciating detail.

The more you dig into the specifics of what we’re talking about here, start breaking this down into tangible concepts without all the acronyms and buzzwords, you start to realize that this really is the telco world embracing everything that OpenStack is all about, but just pushing the envelope a bit with some requirements which are a pretty natural evolution for us, but we might not otherwise have expected to come about for some time yet.

I guess the summary here is that if you're skeptical, that's cool ... you're not alone. But please do take the time to see through the complexity and confusion to the simple fact we're poised to be a key part in turning the telco data-center, and how this is just another exciting part of our goal to "to produce the ubiquitous Open Source Cloud Computing platform".