Mr OpenStack, Tear Down This Tent!

I’ve been an active member of the OpenStack community since 2012. In that time I’ve seen plenty of changes, but none as big as in 2015 when the OpenStack Technical Committee (TC) introduced The Big Tent. This new governance model for OpenStack was intended to solve the following problems:

  1. There was no agreed definition of what should constitute the ‘Integrated Release’
  2. There were too many projects vying to become part of the ‘Integrated Release’

The solution implemented by the TC was to abolish the ‘Integrated Release’ entirely, and effectively open the gates to any project that wanted to officially become a part of OpenStack. This, in my opinion, was a big, big mistake.

The rationale behind this change was noble. OpenStack always strives to be as open and inclusive as possible. Our project and our community are built upon the Four Opens. Allowing all projects to become first class citizens was a huge step forward in that quest for community inclusivity, but it has come at a cost to our primary mission – powering the datacenter. As it’s put on our website:

OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface.

This definition is how I have always thought about OpenStack, and I think that this is how much of the industry views us. Before The Big Tent, anything that didn’t fit within this definition would sit outside of the ‘Integrated Release’. The answer to the question, “What is OpenStack?”, was the ‘Integrated Release’. After The Big Tent, the answer to the question, “What is OpenStack?”, is everything developed within the OpenStack ecosystem. As a list, this looks like:

Sixty individual projects. All OpenStack. Many with very little or no coupling at all. While bringing all of these projects together under The Big Tent has made it easier for developers to start new initiatives, it has made it significantly harder for operators to adopt and maintain OpenStack as the fabric of their datacenter. It has made it harder for customers to assess whether OpenStack is right for them, and it’s made it impossible to simply answer the question, “What is OpenStack?” (just take a look at the patch that’s currently trying to do exactly that). If we are not constantly striving to make all of these things easier, then we are not doing our jobs properly.

OpenStack is always changing, always evolving. Part of this evolution process is being able to take an honest look at the way we work and decide what’s working, and what isn’t. I believe that now is the time to be open and honest in our assessment that The Big Tent has failed to bring the improvements many hoped for. It’s time to move on, and make OpenStack a cohesive project once again.

If I were on the Technical Committee

I’m a believer that there’s little value in pointing out problems without offering solutions, so if it were up to me this is what I would change:

No More Big Tent

I think the reasons for this are well described above.

Provide a clear, narrower definition of OpenStack

We desperately need a consistent, concise answer to the “what is OpenStack?” question. Something that operators can be pointed towards, to download in a single package. To me, this means agreeing on the core set of services that we as a community consider to be the “standard” deployment. As a starting point, let’s take the existing list of core services:

  • Cinder
  • Glance
  • Horizon
  • Keystone
  • Neutron
  • Nova
  • Swift

I’ve added Horizon to this list due to it’s inclusion in the mission statement above. Anything outside of that list is not “OpenStack”. However, there are many excellent projects that compliment this list and are found in many deployments at a variety of scales. Therefore, we need to:

Introduce a new place for complimentary projects to live – The OpenStack Family

The name is just a placeholder, but I think the concept is solid. We absolutely want these projects to continue to be a part of the OpenStack ecosystem. We should continue to provide them with the level of support they’re used to today, and make them an integral part of our summits and promotional material. The list of projects in this category will look something like:

  • Barbican
  • Ceilometer
  • Congress
  • Designate
  • Heat
  • Ironic
  • Magnum
  • Manila
  • Murano
  • Sahara
  • Trove
  • Zaqar

I’m sure that some of these could be argued to fit into the core OpenStack definition by now, so this distinction will need to be made carefully. These two categories will capture all of the projects mature enough to get the OpenStack seal of approval, but new projects still need somewhere to incubate. This is why we should:

Bring back Stackforge!

UPDATE: It’s been pointed out that my understanding of the Stackforge retirement was wrong. Though the Stackforge namespace was retired, the incubation framework remains to this day. I’m going to leave the original section here so others can learn from my mistake, and add a new section about the graduation workflow. Thanks to those who brought this to my attention.

Stackforge was an excellent program for incubating new projects without diluting OpenStack as a whole. It was retired in October 2015, at which point all projects had to move into the OpenStack Big Tent or leave entirely. I think we should bring it back and continue to foster innovation within the community and beyond. Developing a project in Stackforge will be the beginning of the path towards becoming part of OpenStack or the OpenStack Family. Everything that is currently considered part of OpenStack but doesn’t appear in the two lists above, will move into Stackforge.

While I sympathize with the TC’s position that there were too many Stackforge projects to review for graduation into OpenStack, I disagree that the right solution was to have no graduation process at all. I believe that the introduction of the OpenStack Family, together with a clear set of requirements for graduation from Stackforge will help to alleviate the review workload on the TC.

Update the graduation path

With the addition of the OpenStack Family, we’ll need a new path for projects that graduate from StackForge. The way I see it, there will be two types of graduation:

  • StackForge -> OpenStack Family
  • OpenStack Family -> OpenStack

The first will be based upon maturity and conforming with community rules (the Four Opens), the second will generally be based on adoption rate. Graduation into the OpenStack Family will indicate that we as a community are vouching for a given project’s compliance with our rules, and that it shows a degree of maturity/usability. Graduation into the OpenStack core will probably be a much rarer event, as it indicates that a given project has become sufficiently widespread in its use that we consider it to be an integral part of a reference OpenStack deployment. There will likely be extra rules applied to OpenStack core projects that are not applied to the OpenStack Family, such as needing to follow the same release cadence.

This added layer of granularity will help us to much better tell the story about which projects are OpenStack, and which projects enhance OpenStack.

Also note that this does not necessitate separate git namepsaces for the three different project categories (neither did the proposal in the retracted section above, but that may not have been clear). The retirement of the StackForge namespace was a big step forward in managing the lifecycle of a project and it’s not something I’d advocate for reverting.

In Summary

I believe that this proposal, or one derived from it, would go a long way towards providing a tighter focus for OpenStack that has been missing since the introduction of The Big Tent. It’s not final, but it’s a good starting point, and I enthusiastically welcome debate. Please feel free to reach out to me here, on Twitter, or on OpenStack irc (john-davidge), whether you think this is the best idea or the worst.


Changing the face of OpenStack

Rob Quote - censored

For my first proper blog post, I’d like to talk about the culmination of a project that began way back in the summer of 2012, when I was but a lowly intern. Hopefully this will lend some insight into how an experimental project becomes a part of OpenStack.

Along with three classmates of mine from The University of Kent, I was thrown headlong into the then burgeoning OpenStack team at Cisco in Santa Clara, California. Cisco’s involvement at the time with OpenStack was much smaller than it is today, which put us in the enviable position of working closely with some of the team’s most influential members. The OpenStack@Cisco group was, and still is, headed by Lew Tucker – a true leader in the world of cloud computing, and quite probably the best boss you could hope for in the tech industry. If you ever have the chance to hear him speak at an OpenStack summit or similar technology conference, I highly recommend you do so. Lew very quickly recognised the four of us as the inexpensive and low-risk resource that we were, and gave us one simple mission statement for our twelve-month internship – ‘Do Something Cool With OpenStack’.

That ‘Something’ was largely informed by Lew’s experience and interest in network visualisation tools, gained during his time at Oracle. It was also honed and turned most expertly into an executable idea by our mentor, Debo Dutta. Debo is one of those rare geniuses building cool technology for technology’s sake. He seemed to understand from day one exactly the best way to inspire us as undergraduates, while having us work on a project that could bring genuine value to OpenStack as a whole. It started slowly, with steps to help us learn the intricacies of the newly emerging world of OpenStack networking (then called ‘Quantum’), while learning the other technical skills we’d need to build a functioning piece of software. The plan looked something like this:

  1. Learn web development with Rails
  2. Learn virtual networking in OpenStack
  3. ???
  4. Profit

Step 1 – Learn web development with Rails

The first step in Debo’s master plan was to get us building simple web pages with Ruby on Rails. Our first two years at university had taught us incredibly helpful things about web development like how to set up a MySQL database, how to use php to build interactive forms, and other technical skills, but it hadn’t done what it probably should have done – taught us about the modern development frameworks that people actually use to build web applications. This first step was the most straightforward, and we had all mastered the excellent Ruby on Rails Guides after not very long. If you’re looking to get started in web development, I’d say that was a great place to begin.

Step 2 – Learn virtual networking in OpenStack – Project Assemble

So there we were, feeling pretty good about ourselves having mastered the basics of Rails within the first week or two of the internship, when Debo informs us of the next goal he has in mind – build a functioning OpenStack networking API in Rails. At the time, this seemed to be an incredibly daunting task. What we had to do was this:

  • Create a Rails-based web app with an API identical to the OpenStack networking service
  • Have it correctly generate and return all data expected by the API caller (UUIDs, gateway-IPs, etc)
  • Run it on the port usually reserved for Quantum in a Devstack environment
  • Have the API user not be able to tell the difference between the real Quantum and our fake one

This turned out to be a huge amount of fun, and an excellent learning experience. We started this project as complete networking novices, so we learned a whole bunch about CIDRs and other networking basics. After a couple more weeks we had a nearly feature-complete Quantum API running in a Rails-powered web application. Quantum at the time was nowhere near as feature-rich as it’s successor, Neutron, is today, so this task would be much harder if we were starting now!

Project Assemble
A successful response from the Assemble API server. Complete with an ugly green-on-black terminal.

Under Debo’s guidance, Project Assemble was finished quickly and to a standard that impressed Lew enough to have him issue his ‘Do Something Cool’ challenge. We were now about to enter ??? territory.

Step 3 – ??? – Project TrenchRun

By now we had a pretty good handle on finding our way around the OpenStack networking API. We knew what information we could get from it, and also the information that we couldn’t. In this case, we were interested in knowing about the network topology of both the physical and virtual machines that were communicating with each other in our Devstack environment. This was information that we simply couldn’t get from OpenStack alone, so we wrote a simple python script that could sit on all of our VMs, servers, and laptops, and fill a Rails-based web application with information about the open network connections of each machine. We then fed this data into the sigmajs library to build a simple graph of our network. Here’s what it looked like:

A simple network visualised with sigmajs
A simple network visualised with sigmajs

Each node in the above graph represents a machine, either virtual or physical – with the exception of the white ‘External Connections’ node, which represents an IP address not owned by a machine running our python daemon. Where an IP entry exists for a node, a line is drawn between them. The size of each node is determined by the number of network connections they have. ‘The Deathstar’ is the name affectionately given to our first development server – leading to the always amusing game of ‘Who Blew Up The Deathstar?’ whenever the server went down. The loser (or winner?) of that game would buy a crate of beer! We played it as often as we could. The small nodes around it are VMs running in OpenStack Nova. The larger nodes around the ‘External Connections’ node are our laptops.

This was really cool! It suddenly made the confusing lists of network connections into a simple graph that anyone could easily read and quickly figure out which machines could talk to each other. And it led to an interesting challenge from Debo – what if instead of using data about a user-defined network to build a graph, you could use a user-defined graph to build a network?

The graph is the easy part to understand, but to build it you have to go through the difficult part of building the network. If we could create a tool that turned that process on it’s head, that could be a real and valuable contribution to the OpenStack community. It was an ambitious goal, but we were motivated to give it a shot.

Step 4 – Profit – Curvature

The first part of any large project is of course to decide on a project name. Our first project, Assemble, was a reference to the fantastic and timeless film – Anchorman: The Legend of Ron Burgundy. This probably came from our excitement about attending our first OpenStack Summit in San Diego (scholars maintain that the translation was lost hundreds of years ago). Our second project, TrenchRun, came from our love of Star Wars, and the frequency with which The Deathstar was destroyed during its development. For the project that would consume the majority of our internship, we needed a name that sounded like it could be a real product, so we turned to the existing OpenStack user-interface for inspiration. The OpenStack web-based UI is an excellent Django-based project called Horizon. Our goal was to give the user a higher level of abstraction than what was offered in Horizon. What does the view of the horizon become when you go high enough in an aeroplane? The curvature of the Earth. None of us remembers exactly who came up with the name (I’d like to think it was me!), but we all agree on it’s aerospace-y inspiration.

Over the following months Curvature went though many iterations. Our small team of interns was given complete creative control over the project, which was incredibly liberating and often led to late nights in the office working on what had become a passion project for all of us. Every few weeks we would show a new build to Debo or to Lew, and each time they would offer support and suggestions that helped us to keep the project evolving.

Something which took a long time to evolve was our own perception of how far the project could go. We were just a bunch of interns, right? We hadn’t even graduated university yet. What possible contribution could we make to a community as expert and diverse as OpenStack? But Debo, Lew, and others at Cisco constantly persuaded us that we could really have something here, so without expecting anything we submitted a presentation on Curvature for the 2013 OpenStack Summit in Portland. You should have seen our faces when it was accepted! Like many who have had presentations accepted at major technology conferences, our thoughts on the matter went something like this:

  1. Wow! This is amazing! They accepted our talk! This is so exciting!
  2. We have to talk about this for 40 minutes?! How are we going to fill all that time?
  3. What if no one shows up? What if no one likes it?
  4. Why did we submit a presentation? This was the worst idea ever!

So the four of us set to work on something that the stereotypical Computer Science undergraduate is not particularly keen on – writing a presentation to be given in front of an audience. It was hard work, and it was fun. There are few ways better to self-critique your work than by trying to see it through the eyes of someone who’s never seen it before, and trying to anticipate the questions they might ask. In the week or two before the Summit I don’t remember leaving the office before sundown even once. But we weren’t doing it because our boss was telling us to (he wasn’t), we were doing it because we wanted this presentation to go as well as it possibly could, and that felt great.

Our presentation slot was on the very last day of the four day summit, so we had plenty of time to enjoy the sessions and get more and more nervous about our turn as the days went on. When we weren’t in presentations (we were mostly too scared to attend design sessions at this point!) we were at the Cisco booth giving impromptu demos of Curvature to anyone who would listen. Some liked it, some loved it. Maybe someone would come to see us on Thursday after all?

When the time of our presentation finally came, we went to our designated room about 15 minutes early. The room was just starting to empty out after the end of the previous talk, and there weren’t many people staying for ours. As we were being mic’d up there were maybe 20 people in the room waiting for us. Not bad, we thought! 20 people is more than we’d ever presented this to before. But as the minutes counted down to the start of our time slot, people kept coming in and sitting down. Eventually, people were coming in and standing up, because all of the seats had been taken. By the time we started there must have been over a hundred people in the room, but we were so nervous it was difficult to think about that. I won’t say how well the presentation went, I’ll let you judge that for yourself from the official video:

What I will say is that we were completely blown away by the interest in our project from the rest of the OpenStack community. The room was packed, people were taking pictures and videos, and when we were done nervously running through our presentation there was a solid 10 minutes of questions with meaningful insights into what we’d done and where it could go in the future.

OpenStack is a community that is so overwhelming welcoming of new ideas that it turned out to be an absolute joy to present at the Summit. I am constantly thankful for being able to work with the people who make OpenStack one of the most exciting open source projects out there. If you’re wondering whether or not you should submit a presentation about that cool thing you’re working on, the answer is yes!

Step 5 – Back to reality

Shortly after our whirlwind of excitement in Portland we had to return to England to complete the final year of our university studies. It was a bitter-sweet return to a fantastic university, as it meant temporarily leaving the OpenStack community behind to concentrate on graduation. Work continued on Curvature very slowly during this time. It wasn’t until after graduation (and being hired full-time by Cisco!) that Curvature could finally be picked up again to be polished into a truly finished state. For the past year, this work has been done almost entirely by Bradley Jones – the real genius behind Curvature’s D3.js-based visualisations.

Now, finally, after a lot of work with the OpenStack community, Curvature has merged with Horizon as the new Network Topology panel in OpenStack Liberty. A project that started 3 years ago as a learning experience for a group of interns has become the face of virtual networking in one of the most successful cloud computing platforms in the world. I couldn’t be prouder of the work that we started together, and finished so expertly by Brad. None of this would have been possible without our mentors at Cisco, and the wonderful people who make up the OpenStack community.

Next week we’ll be at the OpenStack Summit in Tokyo Japan, talking about Curvature, virtual networking, and other cool parts of cloud computing. If you see us, come and say hi!

OpenStack Summit - Portland
Left to right – John Davidge, Lew Tucker, Jack Fletcher, Sam Betts, Bradley Jones

We are

Sam BettsRob CresswellJohn DavidgeBradley Jones

Special Thanks

Lew TuckerDebo DuttaEdgar MaganaRobert Starmer, Patrick Amor, Brian Bowen, University of Kent Industrial Placement Team, Jack Fletcher

Hello World

“Why haven’t you got a blog, John?!” is a question I have been asked zero times, so I thought it was probably about time to start one. If and when I remember to update this blog it will probably be about the software development work I’m doing in the world of OpenStack, or whatever technology news has piqued my interest at the time. On the off-chance that either of those subjects sounds interesting to you – welcome! I am sorry.