This is the prose version of a talk I gave today at OpenStack Day,
France. The slides are available
Today, I want to speak about a somewhat subtle, meta-topic. I hope to
share some insight on the question of how a company should think about
its investment in any given open-source project. The reason I think
the subject is important right now is that, as OpenStack follows its
inevitable path through the “hype curve”, we are seeing some major
players in the OpenStack ecosystem have recently re-shaped their
investment in the project. I think there is plenty of opportunity for
companies to take a much more realistic view on this subject, and my
hope is that this will lead to a much more sustainable level of
investment in the project.
My views on this question are so heavily influenced by an early
personal experience, that I’m tempted to talk at length about my own
story. But time is short. Very quickly, though, as I was finishing
university, I realized I wanted to focus my career on open-source, and
I was faced with the obvious question of who was going to pay me to
work on open-source? What value would be open-source work bring to my
employer? I spent a lot of time thinking about this, even after I
found myself employed to work full time on open-source.
An interesting thing happened. Because I had spent a good deal of time
thinking about the value of my own open-source work, I was well-placed
to help my employer think about how, where, and why to invest in
open-source projects. It quickly became apparent to me then - and is
still apparent to me now - that this fundamental question is fraught
with difficulty, and most people struggle greatly to answer it.
Investment vs Business Needs
Over the years, I’ve boiled my answer to this simple-sounding question
to something equally simple - your investment in a project should be
directly related to what the business needs from the project. I think
it’s important to frame it this way, because if the business doesn’t
have a good have a good understanding in the value of an investment,
the business is going to be quick to discontinue that investment when
it is looking at how to make the best use of its resources.
What do I mean about an investment that isn’t directly related to the
needs of the business? Let me dig into that a little bit with some
Focus on Features
First, features. Or, as Thierry Carrez puts it, “tactical
often look at the “what do we need from this project” question through
the lens of feature analysis - what are my requirements? What is missing?
The good thing about contributing features is that it is directly
related to business needs. The thing about your primary focus being on
new feature development is it misses the bigger picture. A community
of developers that is focused only on their own new features is not
going to get anywhere. Who is thinking about the long-term future of
the project? Who is looking at feedback from users? Who is merging the
code for these new features?
All of these types of activities are the basic necessities of any
software project. They are the basis through which new features get
added. You must invest in ensuring that this is happening in this
project whose future you care about.
This anti-pattern is about our choice of language, and how it affects
our thinking. People often talk about “donating” code to a
project. Calling it a donation suggests you have a feeling that
the return on your investment is quite intangible. How long will you
continue with these “donations”?
Related, it’s often quite clear that a major motivation for companies
investing in open-source is the recognition they receive in doing
so. Certainly, in OpenStack, with Stackalytics we have spawned an
impressively pointless competition between companies for ranking in
contributor statistics. If you give your employees the goal of
achieving a particular ranking in the contributor statistics, you may
achieve this, but what does that really achieve?
The type of business value that it delivers is essentially short-term
marketing value. We don’t want investment in open-source projects to
be made out marketing budgets, since that’s possibly the least
reliable source of funding!
Not so long ago, when companies were falling over themselves to
demonstrate their commitment to OpenStack, we saw an interesting
phenomenon - the “100% dedicated upstream resource”. These were
individuals and teams at various companies who were employed to
contribute to the project but - as far as I can tell - were
self-directed and deliberately kept isolated from any “downstream”
This is an incredibly alluring opportunity for those involved! The
company is saying that the project is critically important to the
business and whatever you or your team does to help the project be
successful, that’s valuable to the company! Unfortunately, we can see
how these things go in cycles and, at some point, the company will
take a harder look at what this person or team is doing. When that
happens, the likelihood is that the company has very little visibility
into - or understanding of - the work being done and, worse, the
person or team has no experience articulating how the value of the
work is meaningful to the business!
There are some exceptions to this, even within Red Hat. But as a
systematic way of investing in a project … it’s unlikely to be
Finally, the culmination of a number of these anti-patterns - the idea
that companies should "donate" to a non-profit organization like the
OpenStack Foundation, have that organization use the donations to
employ staff who are "100% dedicated upstream resources", and the
value to companies ...? The recognition it receives for being such
Having some technical staff employed by the non-profit is fine. Some
of my favorite people work for the OpenStack Foundation! My preference
would be for Foundation staff to be in facilitation roles. But
certainly, you would do well to avoid this pattern for the majority of
the contributors on a project.
One example of this is the Linux Foundation's Core Infrastructure
Initiative. In the wake of the
OpenSSL Heartbleed vulnerability, the Linux Foundation brought
together funding from a number of companies to invest resources for
key projects like OpenSSL. It's fascinating because here is a project
like OpenSSL that many, many businesses depended on, but few invested
in. What the Linux Foundation has done is positive, but I can't help
feel we have failed as a community if this is the only way to sustain
these projects. You'll notice that Red Hat hasn't joined this
initiative, despite us being supportive of it - we have always
invested in these projects directly, and think that is actually a
healthier model for the future.
Ok, so that's a bunch of anti-patterns ... what you generally
shouldn't do. What should you do? Well, you need to think
strategically. You are choosing to have your business depend on
something you don't control, but you should not allow yourself to feel
powerless about how successful that choice will be.
Think about the future, the long-term. Given the business choice you
are making, for how long will your business success depend on the
success of the project? Indefinitely? What are you doing to ensure
One way to think about that would be a worst-case scenario. The
unimaginable happens, and the rest of our community disappears. You
are left almost completely alone maintaining the project. What you
focus on? Obviously, you wouldn't choose to depend on a project where
there is a possibility of that happening, but the thought exercise
does help you think about your priorities.
If you are determined to ensure the success of the project, you'll
have your own view of what that success looks like. Are you happy to
hope the community will find a direction that fits your needs, without
any input from community leaders that understand the needs of your
At Red Hat, we talk about wearing "two hats". Given a particular
problem, a particular question of direction, you should have the
ability to understand what's good for the project overall and what's
good for Red Hat. And crucially, you should have a way to reconcile
those two views. This is not a zero sum game. Almost always, you can
find a solution that is good for both the project and Red Hat. Why
would Red Hat want a solution that is harmful to the project?
In order to be successful with any technology, you need to have access
to expertise that understands the technology. There is no better
expertise than the authors or maintainers of the project. These are
the people who understand not just how the technology works now, but
how it evolved there, and how things are likely to change in the
future. They understand the pitfalls, the know the cool new tricks.
You can have access to that expertise by having those experts join
your team. They stop being experts pretty quickly if they stop having
time to work on the project, so their upstream responsibilities should
continue to be a significant proportion of their time. But you'd be
amazed at how their presence on the team can help the whole team be
Red Hat's Model
As somewhat of an aside, since this presentation is not a sales pitch,
think about Red Hat's business model, and our proposition to our
customers. Certainly part of the model is that, through our product
subscriptions, the customer is investing in the projects involved and,
by proxy, is safeguarding the future of the project, gaining a level
of influence in the project, and has access to expertise relating to
A Measurable Goal
Recently, on Red Hat's OpenStack team, and thanks to the influence of
Alexis Monville, we've been using the
"Objectives and Key Results"
framework for using well-defined, measurable goals to guide our
teams. We started brainstorming on these OKRs almost a year ago, and
from those early discussions, I wondered “how do we frame an
measurable goal around our investment in upstream?”. What indicator
could we use to know whether we were on the right track, and to ensure
we’d stay on the right track?
Our first thoughts on this was to look at data like our position in
the contributor statistics, or the number of core contributors, PTLs,
or TC members on our team. None of this sat well with us, though,
because we have seen that these type of goals don’t drive the right
behaviors. When we started drafting a vision statement for each goal,
“Teams of engineers with the required motivation, an understanding of
Red Hat's vision, and empowered with the necessary time,
encouragement, and recognition, working upstream to drive and
influence many diverse parts of OpenStack.”
And there it sat for a while, us all lacking ideas on how to measure
this. Quite recently, Russell Bryant and Doug Hellmann hit on a
promising solution. Survey the team with a small set of questions and
use that sentiment of our measure of success with this goal.
The questions that Russell and Doug developed are:
- Does your team have effective input into the features accepted and design decisions made upstream?
- Does your team have effective input into bug fixes and backports made upstream?
- Does your team have effective input into discussions related to processes, schedules, requirements, infrastructure management, and other decisions made by the upstream OpenStack community in general?
- What level of investment does your team make in upstream work?
The allowed answers are “not enough”, “just right”, “too much”, and
“don’t know”. We also had a free-form comments section which is
helping us gain insight into the results.
Notice one important aspect of this - we are asking individuals about
the effectiveness of the investment their team is making
upstream. That reflects the vision above.
We only recently started running this survey, and we will do one every
release cycle. So far, we’ve had over 70 responses, so that’s a pretty
The really interesting judgment call we need to make is what
percentage of “just right” answers do we want to aim for? It would
seem misguided to aim for 100% - do we really think that it’s
important that every individual feels we’re striking the right
balance? We’ve arbitrarily chosen 80% as our target, which means we
feel pretty good about where we’re at, but there’s still opportunities
One thing I’m loving about my job these days is the direct exposure I
have to Red Hat customers who are choosing to invest in
OpenStack. Deciding to build a significant cloud for production use is
a huge decision, and no-one takes it lightly. It’s exciting, and
companies doing this can look forward to a true transformation in how
they think about IT resources. The truly daunting part is the sense of
responsibility that comes with it.
These are long-term choices being made, and they’re being made because
people believe in the future of this project. Red Hat takes the
responsibility seriously by making choices that we hope will ensure
the project has a long, sustainable future.
I had previously posted summaries of most board
but had stopped doing this when I was appointed as Red Hat's
representative on the board. Lately, I've sensed folks at Red Hat
becoming more interested in the workings of the Foundation, so I
figured it might be useful to start doing this again.
The OpenStack Foundation Board of
met for a two hour conference call last week. The usual disclaimer
applies - this my informal recollection of the meeting. It’s not an
Gold Member Applications
The first half of the meeting was given over to an Executive Session,
after which the board voted to approve China Telecom, Inspur, and ZTE
of the Foundation.
This completes the handling of the 7 Gold Member applications that
were presented to the board at the October 24
in Barcelona. 99Cloud, China Mobile, City Networks, and Deutsche
Telecom were approved at that meeting.
For the first time, the Foundation now has reached the maximum of 24
Gold Members. We will
only be able to consider applications for new Gold Members if an
existing member departs, or if we go through a difficult process to
change the limit in our
User Committee Proposal
Next up, Edgar Magana updated the board on some proposed changes to
that was first discussed at the October meeting.
Currently the bylaws describe the User Committee is an "advisory
committee" of three members appointed by the Board and TC, which
prepares regular reports for the Board and TC. The idea with this
proposal is to make the User Committee a parallel entity to the TC,
with its members chosen through an election of Active User
The bylaws changes outline how the User Committee has at least five
members, that elections of Active User Contributors (AUCs) are held
every six months, an affiliation limit for members of the UC, how AUC
status is determined, and more.
The hope is that feedback will be collected as comments in the
and that the Board will vote on the proposal during our December 6
meeting. A vote of the Board is sufficient to enact the changes.
One point of discussion was whether bylaws changes are necessary at
all. The hope when the bylaws were originally drafted was that the
User Committee would have plenty of lattitude to evolve via changes to
made the point that the key change is for the UC to become a parallel
entity to the TC rather than an advisory committee to the Board and
Next, Toby Ford gave a presentation on a topic he had proposed along
with Imad Sousou, Allison Randal, and Mark Baker. Toby introduced the
topic by saying that as any project evolves and matures, it is
important to reflect on competitive threats and the need to evolve the
Toby talked about a wide set of competitive threats to OpenStack from
the container ecosystem (including Kubernetes and Mesos), to other
storage implementations, to various projects in the Telco world
3GPPP), and the obvious challenge of AWS,
Azure, and Google with AWS revenue over $10B.
Toby moved on to talk about "the good and the bad of the Big Tent",
describing the need for a balance between diversity and innovation
versus consolidation and refactoring. Toby made the case that we're
seeing a lot of expansion in the Big Tent, but not consolidation and
refactoring. He expressed particular concern about how and whether
core projects will be able to evolve, especially if the Big Tent does
not allow for competitors to existing projects.
Toby then put on his "AT&T hat" and talked about the challenges for
OpenStack from an AT&T perspective - the need to get to a control
plane that scales to 10k servers, the problem of managing over 100
different production sites, the challenge of more and more happening
"at the edge" with 5G, innovation happening in the networking space
and how it relates to OpenStack, and the unsolved problem of keeping a
production environment current with latest OpenStack releases.
To wrap up his presentation, Toby listed a few "possible
recommendations" the Board could make to the technical community - (1)
allow core projects to have competition within the Big Tent, (2) a
mechanism to incubate new approaches, and (3) a mechanism to
reationalize, clean up, or refactor.
What followed was a pretty lively discussion that covered much ground,
over and beyond the concerns raised by Toby. While the success of
OpenStack in bringing together such a diverse set of interests was
acknowledged, there was a definite frustration that some form of
necessary change is failing to emerge naturally, and whether it falls
on the Board to try to at least try to clearly articulate the problem
at a strategic level.
Jonathan tried to focus the conversation by giving his perspective
that the Big Tent concerns had tied down until "recent comments in the
media", and that is probably a distraction from the concerns people
are expressing about core projects like Nova and Neutron. He was at
pains to say that this isn't about project teams doing bad work - we
continue to make huge progress in each release, and valuable work is
being done. However, we're definitely seeing frustration from some
quarters that it is difficult to influence the community with their
Jonathan warned that talking too much in the abstract creates the risk
that the discussion will go around in circles. Despite this, the
discussion never did go into any particular detail on where we've
witnessed the problems we were discussing. From my own perspective, it
was clear that much of frustration was as a result of how the
Gluon projects have been
received, but we're failing to learn anything significant from those
experiences by not talking about them in detail.
By the end of the discussion, we had agree to collaborate on a
concrete description of specific problem areas, the questions they
raise, and some possible solutions. The hope is that we would complete
this before our December meeting, and we may then plan a longer
meeting (possibly face-to-face, possibly with the TC) to dig into
Derek mentioned "upstream packaging" on this week's packaging
and asked RDO packagers to participate in the upstream discussions. I
thought some more context might be useful.
First, a little history ...
When I first started contributing to OpenStack, it
briefly looked like I would need to
make some Ubuntu packaging updates in order to get a Nova patch landed.
At the Essex design summit a few weeks later, I raged at Monty Taylor
how ridiculous it would be to require a Fedora packager to fix Ubuntu
packaging in order to contribute a patch. I was making the point that
upstream projects should leave packaging to the downstream packaging
maintainers. Upstream CI quickly moved away from using packages after
that summit, and I've heard Monty cite that conversation several times
as why upstream should not get into packaging.
Meanwhile, Dan Prince was running the Smokestack CI system at the time,
which effectively was being treated as OpenStack's first "third party
CI". Interestingly, Smokestack was using packages to do its deployment,
and for a long time Dan was successfully keeping packaging up to date
such that Smokestack could build packages for patches proposed in
And then there's been the persistent interest in "chasing trunk".
Operators who want to practice Continuous Deployment of OpenStack from
trunk. How does packaging fit in that world? Well, the DevOps mantra of
doing development and CI in environments that model your production
environment applies. You should be using packaging as early on in your
pipeline as possible.
My conclusion from all of that is:
- A key part in building a Continuous Delivery pipeline for OpenStack
is to practice continuous package maintenance. You can glibly say
this is "applying a DevOps mindset to package maintenance".
- How awesome would it be if OpenStack had "upstream infrastructure
for downstream package maintainers". In other words, if downstream
package maintainer teams could do their work close to the upstream
project, using upstream infrastructure, without disrupting
I think the work that Derek, Alan, Dan, John, and everyone else has been
doing on Delorean is really helping RDO maintainers figure out how to
practice (1). I first started maintaining Fedora packages for Fedora
Core 2, so IMO what RDO is doing here is really dramatic. It's a very
different way of thinking about package maintenance.
As for (2), this where we get back on topic ...
At a Design Summit session in
Vancouver, the idea
of maintaining packaging using upstream infra really took hold. Thomas
Goirand (aka zigo) proposed the creation of a "distribution packaging"
team and this triggered a healthy
Derek has since pushed a WIP patch showing how RDO packaging could be
There's a clear desire on the part of the Debian and Ubuntu package
maintainers to collaborate on shared packaging, and it sounds like this
goal of further collaboration is one of the primary motivators for
moving their packaging upstream. This makes a lot of sense, given the
shared heritage of Debian and Ubuntu.
The RDO team is enthusiastic about adopting this sort of upstream
workflow, but the Debian/Ubuntu collaboration has added an entirely new
aspect to the conversation. Despite the fact that RDO and SUSE platforms
have little in the way of shared heritage, shouldn't the RDO and SUSE
packaging teams also collaborate, since they both use the RPM format?
And perhaps deb and rpm maintainers should also collaborate to ensure
To my mind, the goal here should be to encourage downstream packaging
teams to work closer to the upstream project, and have downstream
packaging teams collaborate more with upstream developers. This is about
upstream infrastructure for downstream teams, rather than a way to force
collaboration between downstream teams, simply because forced
collaboration rarely works.
For me, what's hugely exciting about all of this is the future prospect
of the package maintainers for different platforms adopting a
"continuous packaging" workflow and working closely with project
developers, to the extent that packaging changes could even be
coordinated with code changes. With its amazing infrastructure,
OpenStack has broken new ground for how open-source projects can
operate. This could be yet another breakthrough, this time demonstrating
how a project's infrastructure can be used to enable an entirely new
level of collaboration between package maintainers and project
This week's launch of
is a good opportunity to think about a simmering debate in the OpenStack
developer community for a while now - what exactly does NFV have to do
with OpenStack, and is it a good thing?
My own “journey” on this started exactly one year ago today when I
visited a local Red Hat partner to talk about OpenStack and, towards the
end of our Q&A, I was asked something like “will OpenStack support
NFV?”. I’d never heard of the term and, when the general idea was
explained, I gave a less than coherent version of “OpenStack implements
an elastic cloud for cattle; this sounds like pets. Sorry”. After the
meeting, the person who asked the question forwarded me an NFV
whitepaper from October 2012 and, glancing through it, most of it went
right over my head and I didn’t see what it had to do with OpenStack.
Since then, Chris Wright has been patiently talking me through this
space and gently trying to get me over my initial skepticism. Chris
would say that our conversations has helped him refine how he explains
the concepts to open-source developers, and I think he really nailed it
in his keynote at the Linux Foundation’s Collaboration summit in April.
In his keynote, Chris talks about the benefits of collaboration in
open-source and walks through all of the various aspects of how the
networking industry is changing, and how open-source is playing a key
part in all of those changes. He covers, and simplifies:
- Taking the current architecture of proprietary, expensive, complex,
difficult to manage forwarding devices (like routers) and how SDN
(Software Defined Networking) aims to “put an API on it”. This is
what’s meant by “disaggregation of the control plane and data
plane” - that forwarding devices become devices which are controlled
by open standards, and allows your distributed system of forwarding
devices to be controlled and automated.
- NFV (Network Function Virtualization) as a shift in the telco
data-center world which embraces many of the lessons that the
elastic infrastructure cloud has taught the IT industry. More on
- Changes in the “data plane” world, where we’re starting to see the
network device market mimic the x86 server market such that these
devices can be “white box” servers running open-source software.
Again that disaggregation word, but this time it’s about
“disaggregation of hardware and software” and how the software part
can be open-source implementations of optimized packet-forwarding
capabilities which we’re used to seeing implemented in expensive and
proprietary hardware appliances.
But let’s focus here on NFV.
I real don’t know much about the telco industry, but what Chris has me
imagining now is data-centers full of proprietary, black-box hardware
appliances which are collectively know as “network functions” or “middle
boxes”. These boxes are used for everything from firewalls, NAT, deep
packet inspect (DPI), the mobile packet core, etc. These are software
applications trapped in hardware. They’re expensive, proprietary, slow
to roll-out, don’t always scale well and are hindering telco service
providers as they attempt to react to a rapidly changing market.
NFV is about completely re-thinking the architecture of these
data-centers. This is the telco industry re-imaging their data centers
as elastic infrastructure clouds running their “network functions” as
virtualized, horizontally scalable applications on these clouds. The
exciting - simply stunning - aspect of all of this for me as an
open-source advocate, is that the telco industry is settling on a
consensus around an architecture involving open-source generally and
Say that again? These huge telcos want to rebuild their entire data
centers with OpenStack and open-source? Yes.
If, like me, you want to see open-source change the IT world into one
where we all embrace the opportunity to collaborate in the open, while
still successfully building building businesses that serve our users’
needs … then this sounds pretty cool, right?
If, like me, you want to see OpenStack as the standard platform from
which many of the worlds’ elastic infrastructure clouds are built ...
then this sounds like a no-brainer, right?
Well, the thing we need to bear in mind is that these applications (i.e.
network functions) are pretty darn specialized. They need to have a high
level of performance, determinism and reliability. But that does not
necessarily mean they are “pets” and missing one of the key points of an
Let’s take the reliability requirement - when these network functions
are implemented as horizontal scale-out applications, they will look to
achieve high levels of reliability in the same way that typical cloud
applications do - with each tier of the application spread across
multiple failure domains, and by spreading application load
horizontally. Telcos will just want to take this further, with faster
and more deterministic response to failures, while also avoiding any
compromise to application's performance. For example, you’ll see a lot
of interest in how instances are scheduled to take to into account
affinity and anti-affinity within an instance group.
The performance requirement is largely about high-performance packet
processing. How to get a packet off the network, into a VM, processed
quickly and back out again on the network. One of the techniques being
pursued is to give VMs direct physical access to the network via SR-IOV
which, in turn, means the compute scheduler needs to know which physical
networks the NICs on each compute node has access to.
The deterministic requirement is about predictable performance. How to
avoid the vagaries of the hypervisor and host OS scheduler affecting
these performance-sensitive applications? You’ll see work around
allowing operators to define flavors, and application owners to define
image properties, which between them control things like vCPU topology,
vCPU to pCPU pinning, the placement of applications in relation to NUMA
nodes and making huge pages available to the applications. Compare to
Amazon’s memory-optimized and compute-optimized flavors, and imagine
this being taken a step further.
Oh, and another requirement you’ll see come up in this space a lot is …
IPv6 everywhere! I’m certainly down with that.
Want to learn more about the work involved? See the OpenStack NFV
team's amazing wiki page
which goes into excruciating detail.
The more you dig into the specifics of what we’re talking about here,
start breaking this down into tangible concepts without all the acronyms
and buzzwords, you start to realize that this really is the telco world
embracing everything that OpenStack is all about, but just pushing the
envelope a bit with some requirements which are a pretty natural
evolution for us, but we might not otherwise have expected to come about
for some time yet.
I guess the summary here is that if you're skeptical, that's cool ...
you're not alone. But please do take the time to see through the
complexity and confusion to the simple fact we're poised to be a key
part in turning the telco data-center, and how this is just another
exciting part of our goal to "to produce the ubiquitous Open Source
Cloud Computing platform".
(This is a prose version of a talk I gave at OpenStack meetups in
Israel and London recently. Apologies for the wordiness.)
In a recent update Jonathan gave to the Board of Directors, we described
how OpenStack has had 2,130 contributors to date and 466 of those are
active on a monthly basis. That’s an incredible statistic. There’s no
doubt OpenStack has managed to attract an unusual number of contributors
and, for such a complex project, made it relatively easy for them to
However, this isn’t just a numbers game. I often hear mutterings that a
much smaller, focused group could achieve the same velocity that
OpenStack is achieving. In some sense that’s true, but I think that the
diversity of interests and priorities is the energy that a community
like OpenStack thrives on.
The question then is how to improve the overall quality of our large
number of contributors. In order to do that, we need to be able to set
expectations. What do we expect and value from our contributors?
What I’m going to attempt to do here is define The Prototypical
OpenStack Developer. The ideal that we should aspire to. The standard
that all contributors should be held to.
(But … bear with me here. I’m being a little tongue-and-cheek.)
Ok. Where do we start? How do we begin to forge this hero from the raw
resources we are presented with?
Let’s start with the basics. The breadth and depth of knowledge you need
on a variety of computing topics.
On virtualization, you could start with KVM. You should know about
CPU extensions such as Intel’s VT-x and I/O virtualization with VT-d and
PCI SR-IOV. Some knowledge of the history of software based
virtualization and paravirtualization would be nice context too. Now
understand the responsibilities of the KVM kernel module versus the
userspace component, qemu. How does qemu emulate various devices? How
does live migration work? How does a hypervisor use page table flags to
track dirty pages during a migration?
And there’s probably little point in understanding all of this without
understanding the x86 architecture in some detail. Understanding how it
compares to RISC architectures would be no harm. Memory segmentation,
MMUs, page tables are all fun topics. You really can’t get into this
without learning a bit of assembly, at least the basic idea. The history
of x86, from real/protected mode to modern day PAE or x86-64 are all
important to understand. Ignore Itanium, though. It’s not enough to just
understand the CPU, though, you need to go beyond and think about how
that CPU interacts with peripherals using DMA and buses like PCI.
And, honestly, if you go this far you may as well understand basic
digital systems theory, like how you can construct a counter or register
from a set of basic logic gates ...
Woah, I think I’ve digressed a little. That’s virtualization. Do the
same for storage and networking. I’ll leave that as an exercise for
That’s just the concept behind the basic resources managed by OpenStack,
though. It’s a pretty complicated distributed system, so it’s pretty
essential you do some reading on that topic. What do terms like “quorum”
and “consensus” mean? Why do people describe the Paxos algorithm as
“quicksort of distributed systems”? What do people mean when they
describe OpenStack as a “shared nothing” architecture, and are they
crazy? How would you describe OpenStack’s approach to fault tolerance?
And obviously related to all of this is the need for deep knowledge of
databases and messaging systems. We seem to have a large number of
ex-MySQL consultants on this project, but don’t let that be an excuse.
You know what foreign keys and cross-table joins are, right? And you
really need to know the kind of operations which will simply lock
individual rows rather than an entire table. For messaging, there’s a
little research you can do there. We’re all about AMQP in OpenStack, but
there’s been a few other messaging protocols in the past. My personal
favorite is CORBA. What’s the difference between a broker, router and
peer-to-peer based architecture? What’s this “fanout” and “topic” things
we talk about in messaging? Incidentally, you know that we’re not
actually using the standard AMQP protocol in OpenStack, right?
You needn’t have touched a line of code at this point. But, if you’re
going to contribute to OpenStack, you need to code, right? Almost
certainly in Python, but we like ourselves a little Bash too. With
Python, it’s important to understand not just the syntax from the most
basic to the more advanced topics like iterators, decorators, context
managers and metaclasses. You also need to have a good knowledge of the
huge number of python libraries out there, inside and outside the core
Python distribution. We need true Pythonistas. Oh, and we’re in the
process of porting to Python 3, so make sure you understand the
differences between Python 2 and 3.
But wait, wait. That’s no good. You can’t just dive straight into
Python. You need to start with C. Allocate and free your own memory,
damnit. You can’t go through life without learning about pointers. Now
learn how to use threads and the various synchronization primitives out
there, and why it’s all just terrible. Now learn about asynchronous I/O
techniques; what an event loop is, how you use select() and non-blocking
sockets to write a single-threaded server which processes requests from
multiple clients. Oh, Richard Stevens. My hero. Don’t be afraid to read
a few RFCs.
Speaking of authors, we forgot algorithms. Yes, those. Just carefully
study all three volumes of Knuth.
Now, before returning to Python, perhaps you should implement a REST API
in Java using JAX-RS and a web UI using Ruby on Rails. Hang out with the
cool kids and port your UI to Sinatra, before realizing that’s not cool
anymore and switching to Node.js.
You might be ready to contribute some code to OpenStack at this point.
But, I hate to think of anyone writing software without having a full
appreciation of the user experience design we’re driving towards. We
don’t want the inmates running the asylum, do we? Which “personas” are
we designing for? “As a web developer, I want to launch a virtual
machine to test my code in.”
Wait, we forgot tools. You can’t get anything done without knowing
your tools. You’re going to do all of your work on Linux, whether that
be in VMs or by running Linux on your main machine. If you’re a serious
person, you need to learn emacs. You’re going to become very close
friends with grep and sed, so learn yourself regular expressions. Lazy
and greedy regexs, both. You know how to do a HTTP POST with curl,
Ah, git! Oh, the glorious git! You can never learn too much about
git. It’s the gift that keeps on giving. If you think I’m joking, spend
some time getting to know interactive rebasing. Reordering, editing,
squashing and splitting commits! Re-writable history! Where have you
been all my life? No git detail is too obscure to ignore. Learn how a
tilde is different from a caret in revision parameters. How you can
delete branches by leaving out the first part of a refspec in a
git-push. Force override, exciting! Is your mind blown yet? No? Find out
how git’s reflog is a history of history!
(Give me a second to calm down, here)
Now, you’ve got to realize something. Based on everything you’ve learned
so far, you could probably write OpenStack on your own. But that’s not
what’s going on here. You’re collaborating. You’re following a
process. How we collaborate and why we follow certain processes is a
more complex, involved and undocumented topic than anything you’ve
learned so far.
To really understand how we get stuff done in OpenStack, you need to be
steeped in open source culture. Understand what we mean when we say
things like “rough consensus and running code” or “do-acry”.
Perhaps start by following the linux-kernel mailing list for a few
months, watching how controversial discussions are worked through and
the subtleties that determine who holds the balance of power and
influence. Don’t worry if you’re shocked and appalled by how unfriendly
it all seems, you’re not the first. If that’s your one take-away from
the kernel, that was time well spent. Now seek out friendlier
communities and understand how they get stuff done. Compare them to
OpenStack and ask yourself questions like “how does our reliance on
voting to make decisions compare to other communities?” or “why does
there seem to be less flamewars in OpenStack than elsewhere?”.
The history of open source is important, will inform how you engage with
OpenStack and that, in turn, will influence how OpenStack evolves. Learn
about the “free software” versus “open source” camps, and how those
philosophies relate to the choice of copyleft licenses like the GPL
versus permissive licenses like Apache, MIT or BSD. Are you in this for
the freedom of users of your code, or are you in it to build
collaborative software development communities? That contributor
agreement you were asked to sign before you contributed to OpenStack -
how do you feel about that?
Think about the different governance models that open-source
communities adopt. Learn about benevolent dictators, project management
committees, “commit bit”, consensus based decision making and the pros
and cons of our representative democracy model.
Learn about the release processes various projects use. Time based
versus feature based. Rapid release cycles with merge windows. Planning
periods, feature freezes, release candidates, stable branches. How do
different distros do this when there are so many maintainers and
packages involved? We use Python a lot, how do they coordinate their
That’s all very well, but it’s important not to be blind to the world
outside open source. Understand how extreme programming and agile
software development evolved. Read the Agile Manifesto. Understand how
this all relates to Continuous Integration, Continuous Delivery and
DevOps. We’re operating in a much different context, but is code review
our variant of XP’s pair programming? Is our gated master superior to
traditional post-commit CI?
You can now consider educated to a basic level. But is that enough to be
an effective contributor? Do you now have everything you need to make an
impact? No, far from it. The hardest part is learning to be a good
human. You need to have superb communication skills, in English of
course, mostly written communication skills for mailing list, gerrit and
IRC discussions. We do meet twice a year in design summits, so you need
to be able to present and defend your ideas in person too. You need to
work on that Irish mumble of yours.
More than that, though, you need to understand people. You need to know
when to be empathetic, when to be pragmatic and when you be dogmatic.
When is someone’s -1 on your patch likely to be an intractable veto and
when is it simply a take-it-or-leave-it suggestion? What fights are
worth fighting? How can you build up kudos points by assisting your
fellow contributors and when is the right time to call in some favours
and spend those kudos points?
Ok, we’re ready to go! How do we put all of this into practice?
Probably the best way to start contributing to the project is by doing
code reviews. You should probably be spending at least a couple of
hours on code review every day. Not just because the number of code
reviewers on a project has the greatest influence on its velocity, but
also because its the best way to start building trust with your fellow
contributors. If you can show yourself as thoughtful, committed and
diligent through your code reviews, then other code reviewers will be
much more inclined to prioritize your patches and less carefully
scrutinize your work.
A good code reviewer manages to simultaneously focus on the little
details while also considering the big picture. Try not to just leave +1
on patches, but instead a little commentary that shows the kind of
things you’ve taken into consideration. Why should anyone trust that
your +1 was the result of 2 hours of careful analysis, research and
testing rather than just 2 minutes of coding style checking?
Also, think about who you are building up trust with. As a new code
reviewer it’s probably more fruitful to provide helpful input on some
meaty patches from some of the lead developers on the project. Then
again, patch triage can be hugely helpful too - catch obvious problems
in patches before the core reviewers ever get to the patch. Don’t forget
to mentor new contributors as a code reviewer, though. Code review is
the face of the project to these contributors and its your opportunity
to show how you can lead by example.
Now, you obviously want to contribute code. Find some gnarly bug to
fix, perhaps some race condition only rarely seen during automated
tests. With all the code reviewing you’ve been doing, you’ve acquired
excellent taste in coding and your work will no doubt live up to those
standards. Don’t forget to write a detailed, helpful commit message and
include a unit test which would catch any regression of the issue. If
this is a more substantial change, you must split your change into
smaller chunks where each patch represents a logical step in your
progression towards the final result.
If you’re making a substantial addition like a new feature or a
re-architecture, you need to document your design in some detail in a
blueprint. Make sure someone reading the spec can quickly understand the
problem you’re trying to solve, why it’s important and the general idea
behind your solution. Then make sure there’s enough background
information included that a reviewers work is made easy. Include the use
cases, any relevant history, related discussions or bugs, alternative
approaches considered and rejected and any security, upgrade,
performance or deployer impact. Describe how your work will be tested
and what documentation changes will be required.
While we’re on the subject of blueprints, don’t forget that these too
need reviewers. Most projects now review the specs associated with
blueprints using gerrit and so this is a way for you to demonstrate your
design skills and catch things which no-one else has yet considered.
Back to code, though. Yes, it’s important to contribute to the various
integrated service projects like Nova, Neutron, Swift and whatnot.
However, there are a bunch of other areas where code contributions are
always needed. For a start, the client projects are always forgotten.
Then there’s the cross-project technical debt that the Oslo program is
hard at work cleaning up. We’re also gradually porting all of OpenStack
to Python 3, and this is going to be a multi year effort requiring the
help of many.
We also place a huge emphasis on automated testing in OpenStack, and the
awesome CI system we have doesn’t come from nowhere. You should always
be ready to jump in a contribute to the infrastructure itself, tools
like devstack-gate, zuul, nodepool or elastic-recheck. And, last but not
least, our functional test suite, Tempest, is always desperately in need
of more contributions to increase our test coverage.
Security is critical in a public-facing service like OpenStack, and
there are several ways you should contribute in this area. Firstly,
there is a small vulnerability management team which collaborates with
each project’s -coresec team to handle privately reported security bugs,
ensuring a fix is prepared for each supported branch before a
coordinated, responsible disclosure of the issue first to vendors and
then the wider world. Important work is this. There’s also a security
group which is trying to bring together the efforts of interested
parties to prepare official notices on security issues that aren’t
actual vulnerabilities, develop a threat analysis process for OpenStack
and maintain the OpenStack Security Guide. They need your help! Most
importantly, though, you need to be security conscious as you write and
review code. There’s a good chance you’ll find and report an existing
vulnerability during the course of your work if you keep your eyes open!
And then there’s docs, always the poor forgotten child of any open
source project. Yet OpenStack has some relatively awesome docs and a
great team developing them. They can never hope to cope with the
workload themselves, though, so they need you to pitch in and help
perfect those docs in your area of expertise.
I mentioned bugs. We must not forget the bugs! Bugs are one way
users can provide valuable contributions to the project, and we must
ensure these contributions are valued so that users will continue to
file bugs. With over 700 configuration options in Nova alone, the
project can’t possibly test all possible combinations by itself so we
rely on our users to test their own use cases and report any issues as
bugs. You should help out here by setting aside some time every day to
triage new bugs, making sure enough information has been provided and
the bug has been appropriately tagged, categorized and prioritized.
Along those same lines, users often struggle with issues with aren’t
obviously or necessarily bugs. You should also pay attention to forums
like ask.openstack.org or the openstack-operators mailing list. Any
outreach you can do to help users be successful with OpenStack will pay
massive dividends in the long run, even just in terms of your
understanding which issues are most important to real users. This
outreach should extend to your attending OpenStack meetups, giving
presentations on your work and listening to what users have to say.
Speaking of mailing lists, we have a hugely active openstack-dev
mailing list, with over 2500 emails in April alone. This is the center
of all activity happening in OpenStack at any time. You really must
track what’s happening there and engage where you can help move things
forward positively. It’s a struggle to keep up, but it really isn’t an
However, one of the side effects of openstack-dev being overloaded is
that many important conversations now happen IRC. You can’t expect
to be around for all of those, so make sure to remain connected and log
all channels so you can catch up later.
Because conversations can be spread around multiple places, it can be
helpful to link all of these conversations with little breadcrumbs. A
mailing list thread might reference a gerrit review, which might
reference a log of an IRC conversation, which might reference a blog
post, which might reference a bug, which might reference a previous
commit message which referenced a previous mailing list thread.
Don’t be fooled into thinking IRC is all about the serious stuff,
though. It’s also a place where you can get to know your fellow
contributors on a personal level and build up yet more of that all
important trust. You will make friends working on OpenStack and some of
those friendships will last longer than your involvement in
OpenStack itself. That’s a hugely positive sign in any community. Beware
of forming cliques, however. We need this community to be open to the
most diverse set of contributors, and not all of those will buy into
US-centric young white male geek humour, for example.
Speaking of cliques, it’s popular to accuse OpenStack developers on
being so self-absorbed that the needs of real operators and users are
ignored. That OpenStack developers aren’t held responsible for the real
world consequences of the decisions they make. “You write code
differently when you carry a pager”. Lorin Hochstein proposed an “Adopt
a Dev” program where operators could invite individual developers to
shadow them for a few days and share their experience in the terms of a
summary, bug reports and blueprints. Basically, you should take any
opportunity you can to get your hands dirty and help operate a
production OpenStack service.
Related to the needs of operators are the deployment, configuration and
operational tools out there which desperately need contributions
with people more familiar with the dirty details of how the software
works. Many developers use devstack to deploy their development clouds,
but there’s huge benefit in occasionally deploying something more
production-like and contributing to whatever tool you used. TripleO is a
great deployment effort to contribute to because it’s attempting to
create a space where everyone interested in deployment can collaborate,
but also because it closely tracks the development branch of OpenStack.
Once you have succeeded at making an impact as an individual
contributor, you should look to extend your leadership efforts
beyond simply leading by example. Naturally, you’ll tend towards
volunteering for the responsibility of the PTL position on whichever
program you contribute most to. To demonstrate your willingness and
trustworthiness for the position, perhaps you’ll suggest the PTL
delegate some of their responsibilities to you.
Your leadership interests should extend beyond a single project too. In
some ways, the kind of cross-project issues considered by the
Technical Committee are as important as the per-project
responsibilities of PTLs. Do you have strong opinions on how, why and
when should add new programs or Integrated projects. If not, why not?
The governance of OpenStack and the shared responsibility for the future
direction of OpenStack extends beyond the TC and PTL’s governance of the
project itself, to the role of the Foundation Board of Directors in
protecting, empowering and promoting the project as well as ensuring
there’s a healthy commercial and non-commercial ecosystem around the
project. Do you care how the TC and board divide their responsibilities?
Or how much explicit corporate influence is appropriate in the technical
decision making of the project? Or how the board makes legal decisions
which directly impact the project? Or how individual members elect their
representatives on the board? You should.
Wait, wait, I’m forgetting a bunch of stuff. You should care deeply
about bringing contributors on board and participate in the awesome OPW
and GSoC programs. It’s important to keep track of how the project is
perceived, so you should read any articles published about the project
and follow even our worst detractors on twitter. Watch carefully how our
major competitors like AWS and GCE are evolving. Make sure to keep on
relevant new developments like NFV or Docker. Keep an eye on new
projects on Stackforge to track how they develop.
Huh, wait. You’re probably employed to work full time on the
project, right? Well, you really need to learn how to wear upstream and
downstream “hats”. You need to understand how you can help your employer
be successful with their objectives around the project. You need to be
able to reconcile any apparent conflicts between your employers’ needs
and the best interests of the project. This is not a zero sum game. Meet
with your employer’s customers and partners, help deliver what OpenStack
product or service your employer is providing, mentor colleagues on how
to successfully engage with the project and be the bridge over the
upstream and downstream gap.
Above all, through all of this, be nice to everyone you encounter and
wear a smile.
BZZZT … BURNOUT ALERT
I’m obviously being facetious, right? There’s no way anyone can possibly
live up to those expectations and live to tell the tale?
It’s pretty obvious when you put it all together like this that these
are unreasonable expectations. The hero of this tale does not exist.
Many of us have tried to be this person, but it’s just not possible.
Read into this, if you like, a very personal tale of burnout caused by
unreasonable self-imposed expectations.
But really, what I want to get across today is that you don’t need to be
this hero in order to contribute. Far from being too many active monthly
contributors, five hundred is just the tip of the iceberg. Why shouldn’t
every attendee of every OpenStack meetup be able to contribute in some
When mentoring new Red Hat engineers, my basic advice is always “find
your niche”. Find something that takes your interest and that you can
see an obvious path towards making a significant impact, and go deep!
Ignore pretty much everything else and do your thing. Maybe after a
while you’ll have got the ball rolling of its own accord and that there
are other areas you can now make an equally big impact on. Or perhaps
you’ll stick with this niche and continue to make an impact doing it
over the longer term.
One of my favorite examples of a less likely niche is bug triage. Back
in the summer of 2001 when I started seriously contributing the GNOME
project and became a maintainer of its CORBA ORB, ORBit, another new
contributor to the project called Luis Villa posted this
Hey, everybody. By way of introduction: I'm the new bugmaster at
Ximian. As some of you may have noticed, I'm slowly moving towards
cleaning out evo and RC bugs from bugzilla.gnome and into
Luis went on to breath new life into GNOME’s “bugsquad”, helped put in
place a highly effective bug triage process and taught the GNOME
community how to truly value and celebrate the contributions of both bug
reporters and bug triagers. If you want to make fame and fortune in the
open source world, how many people would pick bug triage as the place to
start? Well, Luis did and made a huge impact, before moving on to
engineering management and then giving it all up to go to law school. He
is now Assistant General Counsel for the Wikimedia Foundation.
There’s a real “find your niche” lesson in that story, but also a lesson
that we as a community need to learn to truly value and celebrate all of
the myriad of different ways that contributors can help the project.
Rather than judge others based on how they’re not contributing, rather
than feel exasperated when so few others share your passion for a
particular niche no matter how important it seems to you personally, we
as a community need to acquire a greater level of empathy for our fellow
We also need to experiment with ways of running the project so that
different roles and niches are appropriately recognized. Does the focus
we put on PTLs detract from the valuable project management
contributions others make? Are official programs the only way of
recognizing the importance of particular areas? If programs are the only
way, do we need to be more open to creating programs wherever a group of
people have coalesced around some particular effort? Do we need to
explicitly raise the profiles of those contributors doing hard
behind-the-scenes work in areas that we don’t typically recognize? Are
we building a culture that places too much emphasis on recognition and
instead roll back some of the ways we recognize people now?
Lot’s of questions, few answers. But hopefully this can get the