Crusty Blaa

The status quo around RHEL source code availability has changed, as has Red Hat’s stance on RHEL rebuilds. For example, we now say:

Ultimately, we do not find value in a RHEL rebuild and we are not under any obligation to make things easier for rebuilders; this is our call to make. ... Simply rebuilding code, without adding value or changing it in any way, represents a real threat to open source companies everywhere.

If the status quo has changed, doesn’t that mean that our values have changed? I believe the opposite is true - this is Red Hat fully confronting its values, taking uncomfortable steps to make our values more clear, and moving beyond an uneasy ambiguity that I think has existed for too long. And that’s not a convenient, reverse-engineered conclusion to justify the change after-the-fact. I can actually pinpoint the discussion in 2011 where our values on this became clear to me. Let me explain.

I joined Red Hat 20 years ago via the GNOME project. Any personal values I had around software freedom were not particularly strongly held, and GNOME itself had a sometimes fractious relationship with the FSF. But I was drawn in and excited by the potential of developing software in the open, with a community of like-minded contributors with diverse motivations for contributing. This was a time of open-source vs free software debates, and I found myself observing, fascinated, and leaning towards the open-source perspective. To pick one random example of the sort of argument that appealed to me (while absolutely rejected by other friends in the community):

Richard thinks there is a moral imperative underlying the free redistribution of software, and now, by extension, other information. Richard feels that since there isn't any physical cost associated with copying software, limiting free redistribution is a form of extortion. I on the other hand feel that it's immoral to try to compel someone else to give you something they've created without compensating them in some way. That is, when software is freed, it is a gift, not the result of an obligation...

When I joined Red Hat, I again found myself observing the perspectives of my colleagues and the decisions we were making. I guess I was trying to identify Red Hat's values on this knotty subject. It's a funny thing trying to pin down a group’s values - you're unlikely to find a consensus view, and neither are you simply looking for the view of a set of senior leaders.

Over time, we developed a mission statement - “To be the catalyst in communities of customers, contributors, and partners creating better technology the open source way” - and a set of behavioral values - freedom, courage, commitment, and accountability. I found the omission of anything about software freedom is telling. My sense has always been that we are reluctant to open a potentially divisive debate on this, perhaps indicating a nervousness that there is an underlying and unspoken division between community-oriented technical leaders and business-oriented executive leaders.

I happened to be in the Westford office when Oracle announced its RHEL rebuild - Unbreakable Linux - in 2006. Sentiments such as “With this move, Oracle simply rips off Red Hat's mind-share, while promising a cheaper price” were widely shared. There was a sense of existential threat, and a need to rally and respond that RHEL was “Unfakeable Linux”. I 100% bought into this, but I also wondered… isn't this software freedom in action? If our customers aren’t paying for the bits, what's the problem? And yet we did have a problem with this move. It stung. It was unfair. It wasn't in the spirit of this industry movement we thought we were all a part of. And so, it seemed pretty clear to me - this sort of software freedom was not one of our values, and if we would have no qualms about limiting this particular freedom.

Some five years later in 2011, I was involved in deciding which license we would use for an exciting new open-source project we were launching based on a codebase from an acquisition. Given a blank slate in terms of license choice, we spent some time discussing the merits of copyleft vs permissive licenses. In that discussion, we seemed to have no interest in preventing any downstream user from incorporating the codebase into a proprietary product - that was fine by us. But we did agonize over the best strategy for building a community, a community where we might hope one day to not be the largest contributor. We chose the Apache License (v2), and the discussion made something clear to me - we saw no “moral imperative” around software freedom, but we were absolutely committed to building in the open, and forming or joining healthy communities - “the open source way” was an core, unshakeable value.

And so I'm glad we're openly confronting our values on this now, even if this move disappoints any of our friends in the open-source world who have different - but hugely overlapping - values. My personal view is that we are staying true to our values even as we limit the redistribution of our products - while respecting the applicable upstream licenses.

Especially when it is in response to a move - by those software vendors or larger enterprise users who do have the resources to pay for RHEL or to create their own derivative of a community distro - to benefit or profit from the value of RHEL without paying Red Hat.

However, Red Hat remains exceptionally committed to the open-source development model, following the principle of "upstream first", developing our products in the open, building communities, and embracing the diversity of users downstream of any of these open-source projects. CentOS Stream - the recently created upstream of RHEL, a project and community that we now use to develop RHEL - shows an additional level of commitment to open-source community building that I would not have predicted 10 years ago when RHEL development (then downstream of Fedora) was entirely behind Red Hat’s firewall.

I can understand the negative reaction to this change. However, a lot of the commentary is pretty muddled, and I would be happy to see a deeper discussion of open-source values. I think that Red Hatters should be proud of our values, and embrace the debate that we have now triggered.

Watching #heartbleed (aka CVE-2014-0160) fly by in my twitter stream this week, I keep wishing we could all just pause time for a couple of weeks and properly reflect on all the angles here.

Some of the things I'd love to have more time to dig into:

• Obviously, the bug itself and its fix
• Why countermeasures in the kernel, libc or the compiler didn't prevent this, or how static analysis tools didn't catch it. (Update: see here and here)
• The protocol design - some wonder why a seemingly application-level feature is included in such a critical security protocol and why the feature is available before authentication has completed. Interesting to note that one of the stated purposes of the feature is PMTU discovery for DTLS yet it's enabled for plain TLS too
• That it's not only servers which are vulnerable - clients equally vulnerable
• The fact that attacks can't be detected because there is no logging of this failure condition
• The health of the OpenSSL project, talk of its developers being irresponsible when it comes to their development practice
• How the bug was introduced, how much code review or testing was involved. (Update: comments from the original author of the code)
• The realization that few in our industry are actively investing in this project which so many place so much trust in
• The question of whether OpenSSL is the library everyone should be relying on, whether it should be e.g. NSS or GnuTLS
• How something like OpenSSL can be a critical part of a web service's architecture even where e.g. the service is implemented with technologies like .NET
• The question of whether we're putting all our eggs in one basket if an SSL vulnerability can have such far reaching implications
• Just how much was exposed by the vulnerability - e.g. private keys, usernames and passwords, other sensitive data - and how much data could have been compromised by a determined, well-resourced attacker in possession of this knowledge before it was patched
• The efficacy of certificate revocation measures in protecting users from potentially compromised certificates
• How Perfect Forward Security is an important mitigation measure for vulnerabilities like this
• The claim by one site that they could detect attacks in their logs, that there was someone exploiting the vulnerability two weeks before it was disclosed publicly. (Thierry Carrez points out the update to this post which says "other tools [..] can produce the same pattern in the SeaCat server log")
• How our perception of a vulnerability like this has changed now that we know just how aggressive intelligence agencies (the NSA and others) are in their approach to population surveillance and monitoring. This FOSDEM talk is rather prescient in describing OpenSSL as the "crown jewels" for the NSA. (Update: claims that the NSA knew of the bug soon after its introduction and exploited it)
• The approach that was taken to disclosing the vulnerability, how responsible a disclosure process it was and the pros/cons of delaying disclosure further. Apparently CloudFlare was disclosed before the public disclosure, whereas vendors like Red Hat weren't. (Update: more details on the disclosure timeline from Kurt Seifried) (Another update: OpenSSL's official view of the timeline) (Yet another update: interview with Codenomicon's David Chartier)
• How effective entities like Linux distros were in helping getting the vulnerability quickly patched in the wild (see e.g. that Stripe went and built their own patched package rather than wait for an Ubuntu package)
• How much more difficult patching this was due to applications bundling OpenSSL and how much worse it could have been if the practice of bundling was more widespread
• The impact that the impressive marketing - i.e. a cool name vs simply a CVE number, a logo, clear technical writing which speaks to the business impact, etc. - had on the speed with which the patch was deployed

Back in November 2009, ajax sent an email on IRC etiquette to Red Hat's company-wide mailing list. I've had to refer several people to it over the years, so I asked ajax for permission to publish it. He agreed. Here it is in all its glory.

From: Adam Jackson
To: memo-list
Subject: On "ping" etiquitte
Date: Tue, 17 Nov 2009 12:21:30 -0500

IRC has developed a "ping" convention for getting someone's attention. It works because most clients will highlight channels in which your name has been mentioned, so something like

ajax: ping

will make that channel show up pink instead of white for me [1].

I wish to correct, or at least amend, this behaviour. The naked ping should be Considered Harmful, for at least two reasons. The first is that it conveys no information. The recipient of your ping, like you, is a Busy Person. They may be in the middle of something requiring intricate thought, and should not be interrupted for anything less than fire, flood, or six figures of revenue. Worse, _you_ may forget why you pinged someone; when, four hours later, your victim gets back to IRC and responds to you, _you_ will be disrupted in turn trying to remember what was on your mind in the first place.

The second, more subtle reason proceeds from the first. A ping with no data is essentially a command. It's passive-aggressive; it implies that the recipient's time is less valuable than yours. [2] The pingee will respond in one (or both) of two ways. Either they will experience increased stress due to increased unpredictable demands on their time, or they will simply ignore naked pings.

The fundamental issue here is a misunderstanding of the medium. IRC is not a telephone. It's volatile storage. The whole reason the ping works is because the client remembers seeing the ping, and can save it in your history buffer so you can see who was talking to you and why.

The naked ping removes this context.

Please. Save your time. Save my time. Make all of our lives more efficient and less stressful. Ping with data. At a minimum:

ajax: ping re bz 534027

See the difference? Now you've turned slow, lockstep, PIO-like interaction into smooth pipelineable DMA. It's good for your hardware, and it's good for you.

[1] - irssi 4 life.

[2] - Their time may well be less valuable than yours. That's not the
point.

• ajax

tl;dr - the affiliation limit means that at most one of the two Red Hat affiliated candidates can be elected. The cumulative voting system makes it likely that both of us running seriously damages both of our chances of being elected. A preferential voting system like Condorcet or STV would not have this problem.

--

At Red Hat, those of us who contribute to OpenStack take very seriously our responsibility to put what's good for the project first and foremost in our minds - to wear our "upstream hat", as we like to say. That's especially true for me and Russell Bryant.

However, now that the candidate list for the 2014 OpenStack Individual Member Director Elections has been finalized, we find ourselves wrestling with the fact that Russell and I are both running as candidates. Two aspects of our election system make this a problem. First, the cumulative voting system means that those who would be happy to vote for either me or Russell are forced to choose between us - essentially, we are damaging each others' chances of being elected. Secondly, the affiliation limit means that even if we were both lucky enough to receive enough votes to be elected, one of us would be eliminated by the limit.

The combination of these two issues means that we have to factor our affiliation into our decision. The rules place affiliation front and centre in election system, even though Individual Member Directors are not elected to represent their employer.

Now, I'm personally guilty of not pushing this election system issue hard enough over this past year. At one point I favoured experimenting with a tweak to the cumulative system over a more dramatic change because I found the prospect of getting a majority of over 25% of our enormous electorate to vote in favour of a change so daunting. I want to be completely open about this decision we now face because I want to help raise awareness about how important an issue it is.

Given that Red Hat is a Platinum Member and has a automatic seat on the board, the options we're weighing up are:

1. Continue with both Russell and me on the ballot, accepting the risk that we're damaging each others chances.
2. I or Russell remove ourselves from the ballot, giving the other of us the best possible chance of being elected.
3. Brian Stevens steps down from the board and I or Russell takes his place, giving whichever of us remains on the ballot the best possible chance of being elected.

It's not an easy decision. We both feel we have something to offer on the board. Both of us would be very proud to be elected to represent the Individual Members. Both of us feel that Brian Stevens (our CTO who we greatly respect) is the best possible representative for Red Hat on the board.

We will make a decision on this before the election, but right now we don't see any of the options as being particularly better than the others. But, at the very least, I hope everyone will find this useful as a concrete example of why our election system needs to change.

Today, Eoghan Glynn and I announced the first milestone release of a REST API for Red Hat Enterprise Virtualization Manager.

The only current API for RHEV-M is a Windows Powershell plugin which provides a perfectly fine scripting interface for RHEV-M on Windows, but isn't so easy to call remotely or to integrate with another application. By adding a REST API, we're adding an integration interface which we hope everyone will find convenient to use.

If you have a 2.2 installation of RHEV-M, it's a quick and painless process to download our distribution, deploy the WAR to an Java EE application server (e.g. JBoss EAP or AS) and play around with our Apache Felix Karaf based shell. You can also read the API reference guide and jump on our mailing list to give us feedback.

RHEV-M isn't yet an open-source project, so we've had to put a lot of effort into making it possible to develop this REST API in the open. We've concentrated our initial efforts on API design and building a prototype implementation of the API on top of the Powershell interface in 2.2. However, in time for the next release of RHEV-M, our plan is to add a compatible implementation of the API directly to the RHEV-M backend. At that time, the API will be a fully supported part of the product.

If you care about integrating with RHEV-M, now is the time to get involved with the API project. While the API is already quite well defined, there is buckets of scope for design changes and adding new features. And, most of all, we want your help!