Crusty Blaa

Derek mentioned "upstream packaging" on this week's packaging meeting and asked RDO packagers to participate in the upstream discussions. I thought some more context might be useful.

First, a little history ...

When I first started contributing to OpenStack, it briefly looked like I would need to make some Ubuntu packaging updates in order to get a Nova patch landed. At the Essex design summit a few weeks later, I raged at Monty Taylor how ridiculous it would be to require a Fedora packager to fix Ubuntu packaging in order to contribute a patch. I was making the point that upstream projects should leave packaging to the downstream packaging maintainers. Upstream CI quickly moved away from using packages after that summit, and I've heard Monty cite that conversation several times as why upstream should not get into packaging.

Meanwhile, Dan Prince was running the Smokestack CI system at the time, which effectively was being treated as OpenStack's first "third party CI". Interestingly, Smokestack was using packages to do its deployment, and for a long time Dan was successfully keeping packaging up to date such that Smokestack could build packages for patches proposed in gerrit.

And then there's been the persistent interest in "chasing trunk". Operators who want to practice Continuous Deployment of OpenStack from trunk. How does packaging fit in that world? Well, the DevOps mantra of doing development and CI in environments that model your production environment applies. You should be using packaging as early on in your pipeline as possible.

My conclusion from all of that is:

1. A key part in building a Continuous Delivery pipeline for OpenStack is to practice continuous package maintenance. You can glibly say this is "applying a DevOps mindset to package maintenance".
2. How awesome would it be if OpenStack had "upstream infrastructure for downstream package maintainers". In other words, if downstream package maintainer teams could do their work close to the upstream project, using upstream infrastructure, without disrupting upstream development.

I think the work that Derek, Alan, Dan, John, and everyone else has been doing on Delorean is really helping RDO maintainers figure out how to practice (1). I first started maintaining Fedora packages for Fedora Core 2, so IMO what RDO is doing here is really dramatic. It's a very different way of thinking about package maintenance.

As for (2), this where we get back on topic ...

At a Design Summit session in Vancouver, the idea of maintaining packaging using upstream infra really took hold. Thomas Goirand (aka zigo) proposed the creation of a "distribution packaging" team and this triggered a healthy debate on openstack-dev. Derek has since pushed a WIP patch showing how RDO packaging could be imported.

There's a clear desire on the part of the Debian and Ubuntu package maintainers to collaborate on shared packaging, and it sounds like this goal of further collaboration is one of the primary motivators for moving their packaging upstream. This makes a lot of sense, given the shared heritage of Debian and Ubuntu.

The RDO team is enthusiastic about adopting this sort of upstream workflow, but the Debian/Ubuntu collaboration has added an entirely new aspect to the conversation. Despite the fact that RDO and SUSE platforms have little in the way of shared heritage, shouldn't the RDO and SUSE packaging teams also collaborate, since they both use the RPM format? And perhaps deb and rpm maintainers should also collaborate to ensure consistency?

To my mind, the goal here should be to encourage downstream packaging teams to work closer to the upstream project, and have downstream packaging teams collaborate more with upstream developers. This is about upstream infrastructure for downstream teams, rather than a way to force collaboration between downstream teams, simply because forced collaboration rarely works.

For me, what's hugely exciting about all of this is the future prospect of the package maintainers for different platforms adopting a "continuous packaging" workflow and working closely with project developers, to the extent that packaging changes could even be coordinated with code changes. With its amazing infrastructure, OpenStack has broken new ground for how open-source projects can operate. This could be yet another breakthrough, this time demonstrating how a project's infrastructure can be used to enable an entirely new level of collaboration between package maintainers and project developers.

Watching #heartbleed (aka CVE-2014-0160) fly by in my twitter stream this week, I keep wishing we could all just pause time for a couple of weeks and properly reflect on all the angles here.

Some of the things I'd love to have more time to dig into:

• Obviously, the bug itself and its fix
• Why countermeasures in the kernel, libc or the compiler didn't prevent this, or how static analysis tools didn't catch it. (Update: see here and here)
• The protocol design - some wonder why a seemingly application-level feature is included in such a critical security protocol and why the feature is available before authentication has completed. Interesting to note that one of the stated purposes of the feature is PMTU discovery for DTLS yet it's enabled for plain TLS too
• That it's not only servers which are vulnerable - clients equally vulnerable
• The fact that attacks can't be detected because there is no logging of this failure condition
• The health of the OpenSSL project, talk of its developers being irresponsible when it comes to their development practice
• How the bug was introduced, how much code review or testing was involved. (Update: comments from the original author of the code)
• The realization that few in our industry are actively investing in this project which so many place so much trust in
• The question of whether OpenSSL is the library everyone should be relying on, whether it should be e.g. NSS or GnuTLS
• How something like OpenSSL can be a critical part of a web service's architecture even where e.g. the service is implemented with technologies like .NET
• The question of whether we're putting all our eggs in one basket if an SSL vulnerability can have such far reaching implications
• Just how much was exposed by the vulnerability - e.g. private keys, usernames and passwords, other sensitive data - and how much data could have been compromised by a determined, well-resourced attacker in possession of this knowledge before it was patched
• The efficacy of certificate revocation measures in protecting users from potentially compromised certificates
• How Perfect Forward Security is an important mitigation measure for vulnerabilities like this
• The claim by one site that they could detect attacks in their logs, that there was someone exploiting the vulnerability two weeks before it was disclosed publicly. (Thierry Carrez points out the update to this post which says "other tools [..] can produce the same pattern in the SeaCat server log")
• How our perception of a vulnerability like this has changed now that we know just how aggressive intelligence agencies (the NSA and others) are in their approach to population surveillance and monitoring. This FOSDEM talk is rather prescient in describing OpenSSL as the "crown jewels" for the NSA. (Update: claims that the NSA knew of the bug soon after its introduction and exploited it)
• The approach that was taken to disclosing the vulnerability, how responsible a disclosure process it was and the pros/cons of delaying disclosure further. Apparently CloudFlare was disclosed before the public disclosure, whereas vendors like Red Hat weren't. (Update: more details on the disclosure timeline from Kurt Seifried) (Another update: OpenSSL's official view of the timeline) (Yet another update: interview with Codenomicon's David Chartier)
• How effective entities like Linux distros were in helping getting the vulnerability quickly patched in the wild (see e.g. that Stripe went and built their own patched package rather than wait for an Ubuntu package)
• How much more difficult patching this was due to applications bundling OpenSSL and how much worse it could have been if the practice of bundling was more widespread
• The impact that the impressive marketing - i.e. a cool name vs simply a CVE number, a logo, clear technical writing which speaks to the business impact, etc. - had on the speed with which the patch was deployed

Back in November 2009, ajax sent an email on IRC etiquette to Red Hat's company-wide mailing list. I've had to refer several people to it over the years, so I asked ajax for permission to publish it. He agreed. Here it is in all its glory.

From: Adam Jackson
To: memo-list
Subject: On "ping" etiquitte
Date: Tue, 17 Nov 2009 12:21:30 -0500

IRC has developed a "ping" convention for getting someone's attention. It works because most clients will highlight channels in which your name has been mentioned, so something like

ajax: ping

will make that channel show up pink instead of white for me [1].

I wish to correct, or at least amend, this behaviour. The naked ping should be Considered Harmful, for at least two reasons. The first is that it conveys no information. The recipient of your ping, like you, is a Busy Person. They may be in the middle of something requiring intricate thought, and should not be interrupted for anything less than fire, flood, or six figures of revenue. Worse, _you_ may forget why you pinged someone; when, four hours later, your victim gets back to IRC and responds to you, _you_ will be disrupted in turn trying to remember what was on your mind in the first place.

The second, more subtle reason proceeds from the first. A ping with no data is essentially a command. It's passive-aggressive; it implies that the recipient's time is less valuable than yours. [2] The pingee will respond in one (or both) of two ways. Either they will experience increased stress due to increased unpredictable demands on their time, or they will simply ignore naked pings.

The fundamental issue here is a misunderstanding of the medium. IRC is not a telephone. It's volatile storage. The whole reason the ping works is because the client remembers seeing the ping, and can save it in your history buffer so you can see who was talking to you and why.

The naked ping removes this context.

Please. Save your time. Save my time. Make all of our lives more efficient and less stressful. Ping with data. At a minimum:

ajax: ping re bz 534027

See the difference? Now you've turned slow, lockstep, PIO-like interaction into smooth pipelineable DMA. It's good for your hardware, and it's good for you.

[1] - irssi 4 life.

[2] - Their time may well be less valuable than yours. That's not the
point.

• ajax

We're holding a test day for Fedora 16 OpenStack today on #fedora-test-day.

I've been around open source mailing lists for so long now that I tend to forget that our conventions around sending plain text emails, quoting replies, threading, etc. aren't necessarily obvious to everyone out there.

That's especially true for some of the folks working on RHEV-M, many of whom have come from a Windows development background. I eventually realized that saying "just do what everyone else does" wasn't really good enough and spent some time documenting what all those little conventions are.

I've posted some guidelines to the rhevm-api wiki. What am I missing?