Category: openstack

This week’s launch of OPNFV is a good opportunity to think about a simmering debate in the OpenStack developer community for a while now – what exactly does NFV have to do with OpenStack, and is it a good thing?

My own “journey” on this started exactly one year ago today when I visited a local Red Hat partner to talk about OpenStack and, towards the end of our Q&A, I was asked something like “will OpenStack support NFV?”. I’d never heard of the term and, when the general idea was explained, I gave a less than coherent version of “OpenStack implements an elastic cloud for cattle; this sounds like pets. Sorry”. After the meeting, the person who asked the question forwarded me an NFV whitepaper from October 2012 and, glancing through it, most of it went right over my head and I didn’t see what it had to do with OpenStack.

Since then, Chris Wright has been patiently talking me through this space and gently trying to get me over my initial skepticism. Chris would say that our conversations has helped him refine how he explains the concepts to open-source developers, and I think he really nailed it in his keynote at the Linux Foundation’s Collaboration summit in April.

In his keynote, Chris talks about the benefits of collaboration in open-source and walks through all of the various aspects of how the networking industry is changing, and how open-source is playing a key part in all of those changes. He covers, and simplifies:

• Taking the current architecture of proprietary, expensive, complex, difficult to manage forwarding devices (like routers) and how SDN (Software Defined Networking) aims to “put an API on it”. This is what’s meant by “disaggregation of the control plane and data plane” – that forwarding devices become devices which are controlled by open standards, and allows your distributed system of forwarding devices to be controlled and automated.
• NFV (Network Function Virtualization) as a shift in the telco data-center world which embraces many of the lessons that the elastic infrastructure cloud has taught the IT industry. More on that below.
• Changes in the “data plane” world, where we’re starting to see the network device market mimic the x86 server market such that these devices can be “white box” servers running open-source software. Again that disaggregation word, but this time it’s about “disaggregation of hardware and software” and how the software part can be open-source implementations of optimized packet-forwarding capabilities which we’re used to seeing implemented in expensive and proprietary hardware appliances.

But let’s focus here on NFV.

I real don’t know much about the telco industry, but what Chris has me imagining now is data-centers full of proprietary, black-box hardware appliances which are collectively know as “network functions” or “middle boxes”. These boxes are used for everything from firewalls, NAT, deep packet inspect (DPI), the mobile packet core, etc. These are software applications trapped in hardware. They’re expensive, proprietary, slow to roll-out, don’t always scale well and are hindering telco service providers as they attempt to react to a rapidly changing market.

NFV is about completely re-thinking the architecture of these data-centers. This is the telco industry re-imaging their data centers as elastic infrastructure clouds running their “network functions” as virtualized, horizontally scalable applications on these clouds. The exciting – simply stunning – aspect of all of this for me as an open-source advocate, is that the telco industry is settling on a consensus around an architecture involving open-source generally and OpenStack specifically.

Say that again? These huge telcos want to rebuild their entire data centers with OpenStack and open-source? Yes.

If, like me, you want to see open-source change the IT world into one where we all embrace the opportunity to collaborate in the open, while still successfully building building businesses that serve our users’ needs … then this sounds pretty cool, right?

If, like me, you want to see OpenStack as the standard platform from which many of the worlds’ elastic infrastructure clouds are built … then this sounds like a no-brainer, right?

Well, the thing we need to bear in mind is that these applications (i.e. network functions) are pretty darn specialized. They need to have a high level of performance, determinism and reliability. But that does not necessarily mean they are “pets” and missing one of the key points of an elastic cloud.

Let’s take the reliability requirement – when these network functions are implemented as horizontal scale-out applications, they will look to achieve high levels of reliability in the same way that typical cloud applications do – with each tier of the application spread across multiple failure domains, and by spreading application load horizontally. Telcos will just want to take this further, with faster and more deterministic response to failures, while also avoiding any compromise to application’s performance. For example, you’ll see a lot of interest in how instances are scheduled to take to into account affinity and anti-affinity within an instance group.

The performance requirement is largely about high-performance packet processing. How to get a packet off the network, into a VM, processed quickly and back out again on the network. One of the techniques being pursued is to give VMs direct physical access to the network via SR-IOV which, in turn, means the compute scheduler needs to know which physical networks the NICs on each compute node has access to.

The deterministic requirement is about predictable performance. How to avoid the vagaries of the hypervisor and host OS scheduler affecting these performance-sensitive applications? You’ll see work around allowing operators to define flavors, and application owners to define image properties, which between them control things like vCPU topology, vCPU to pCPU pinning, the placement of applications in relation to NUMA nodes and making huge pages available to the applications. Compare to Amazon’s memory-optimized and compute-optimized flavors, and imagine this being taken a step further.

Oh, and another requirement you’ll see come up in this space a lot is … IPv6 everywhere! I’m certainly down with that.

Want to learn more about the work involved? See the OpenStack NFV team’s amazing wiki page which goes into excruciating detail.

The more you dig into the specifics of what we’re talking about here, start breaking this down into tangible concepts without all the acronyms and buzzwords, you start to realize that this really is the telco world embracing everything that OpenStack is all about, but just pushing the envelope a bit with some requirements which are a pretty natural evolution for us, but we might not otherwise have expected to come about for some time yet.

I guess the summary here is that if you’re skeptical, that’s cool … you’re not alone. But please do take the time to see through the complexity and confusion to the simple fact we’re poised to be a key part in turning the telco data-center, and how this is just another exciting part of our goal to “to produce the ubiquitous Open Source Cloud Computing platform”.

(This is a prose version of a talk I gave at OpenStack meetups in Israel and London recently. Apologies for the wordiness.)

In a recent update Jonathan gave to the Board of Directors, we described how OpenStack has had 2,130 contributors to date and 466 of those are active on a monthly basis. That’s an incredible statistic. There’s no doubt OpenStack has managed to attract an unusual number of contributors and, for such a complex project, made it relatively easy for them to contribute.

However, this isn’t just a numbers game. I often hear mutterings that a much smaller, focused group could achieve the same velocity that OpenStack is achieving. In some sense that’s true, but I think that the diversity of interests and priorities is the energy that a community like OpenStack thrives on.

The question then is how to improve the overall quality of our large number of contributors. In order to do that, we need to be able to set expectations. What do we expect and value from our contributors?

What I’m going to attempt to do here is define The Prototypical OpenStack Developer. The ideal that we should aspire to. The standard that all contributors should be held to.

(But … bear with me here. I’m being a little tongue-and-cheek.)

Ok. Where do we start? How do we begin to forge this hero from the raw resources we are presented with?

Let’s start with the basics. The breadth and depth of knowledge you need on a variety of computing topics.

On virtualization, you could start with KVM. You should know about CPU extensions such as Intel’s VT-x and I/O virtualization with VT-d and PCI SR-IOV. Some knowledge of the history of software based virtualization and paravirtualization would be nice context too. Now understand the responsibilities of the KVM kernel module versus the userspace component, qemu. How does qemu emulate various devices? How does live migration work? How does a hypervisor use page table flags to track dirty pages during a migration?

And there’s probably little point in understanding all of this without understanding the x86 architecture in some detail. Understanding how it compares to RISC architectures would be no harm. Memory segmentation, MMUs, page tables are all fun topics. You really can’t get into this without learning a bit of assembly, at least the basic idea. The history of x86, from real/protected mode to modern day PAE or x86-64 are all important to understand. Ignore Itanium, though. It’s not enough to just understand the CPU, though, you need to go beyond and think about how that CPU interacts with peripherals using DMA and buses like PCI.

And, honestly, if you go this far you may as well understand basic digital systems theory, like how you can construct a counter or register from a set of basic logic gates …

Woah, I think I’ve digressed a little. That’s virtualization. Do the same for storage and networking. I’ll leave that as an exercise for the reader.

That’s just the concept behind the basic resources managed by OpenStack, though. It’s a pretty complicated distributed system, so it’s pretty essential you do some reading on that topic. What do terms like “quorum” and “consensus” mean? Why do people describe the Paxos algorithm as “quicksort of distributed systems”? What do people mean when they describe OpenStack as a “shared nothing” architecture, and are they crazy? How would you describe OpenStack’s approach to fault tolerance?

And obviously related to all of this is the need for deep knowledge of databases and messaging systems. We seem to have a large number of ex-MySQL consultants on this project, but don’t let that be an excuse. You know what foreign keys and cross-table joins are, right? And you really need to know the kind of operations which will simply lock individual rows rather than an entire table. For messaging, there’s a little research you can do there. We’re all about AMQP in OpenStack, but there’s been a few other messaging protocols in the past. My personal favorite is CORBA. What’s the difference between a broker, router and peer-to-peer based architecture? What’s this “fanout” and “topic” things we talk about in messaging? Incidentally, you know that we’re not actually using the standard AMQP protocol in OpenStack, right?

You needn’t have touched a line of code at this point. But, if you’re going to contribute to OpenStack, you need to code, right? Almost certainly in Python, but we like ourselves a little Bash too. With Python, it’s important to understand not just the syntax from the most basic to the more advanced topics like iterators, decorators, context managers and metaclasses. You also need to have a good knowledge of the huge number of python libraries out there, inside and outside the core Python distribution. We need true Pythonistas. Oh, and we’re in the process of porting to Python 3, so make sure you understand the differences between Python 2 and 3.

But wait, wait. That’s no good. You can’t just dive straight into Python. You need to start with C. Allocate and free your own memory, damnit. You can’t go through life without learning about pointers. Now learn how to use threads and the various synchronization primitives out there, and why it’s all just terrible. Now learn about asynchronous I/O techniques; what an event loop is, how you use select() and non-blocking sockets to write a single-threaded server which processes requests from multiple clients. Oh, Richard Stevens. My hero. Don’t be afraid to read a few RFCs.

Speaking of authors, we forgot algorithms. Yes, those. Just carefully study all three volumes of Knuth.

Now, before returning to Python, perhaps you should implement a REST API in Java using JAX-RS and a web UI using Ruby on Rails. Hang out with the cool kids and port your UI to Sinatra, before realizing that’s not cool anymore and switching to Node.js.

You might be ready to contribute some code to OpenStack at this point. But, I hate to think of anyone writing software without having a full appreciation of the user experience design we’re driving towards. We don’t want the inmates running the asylum, do we? Which “personas” are we designing for? “As a web developer, I want to launch a virtual machine to test my code in.”

Wait, we forgot tools. You can’t get anything done without knowing your tools. You’re going to do all of your work on Linux, whether that be in VMs or by running Linux on your main machine. If you’re a serious person, you need to learn emacs. You’re going to become very close friends with grep and sed, so learn yourself regular expressions. Lazy and greedy regexs, both. You know how to do a HTTP POST with curl, right?

Ah, git! Oh, the glorious git! You can never learn too much about git. It’s the gift that keeps on giving. If you think I’m joking, spend some time getting to know interactive rebasing. Reordering, editing, squashing and splitting commits! Re-writable history! Where have you been all my life? No git detail is too obscure to ignore. Learn how a tilde is different from a caret in revision parameters. How you can delete branches by leaving out the first part of a refspec in a git-push. Force override, exciting! Is your mind blown yet? No? Find out how git’s reflog is a history of history!

(Give me a second to calm down, here)

Now, you’ve got to realize something. Based on everything you’ve learned so far, you could probably write OpenStack on your own. But that’s not what’s going on here. You’re collaborating. You’re following a process. How we collaborate and why we follow certain processes is a more complex, involved and undocumented topic than anything you’ve learned so far.

To really understand how we get stuff done in OpenStack, you need to be steeped in open source culture. Understand what we mean when we say things like “rough consensus and running code” or “do-acry”.

Perhaps start by following the linux-kernel mailing list for a few months, watching how controversial discussions are worked through and the subtleties that determine who holds the balance of power and influence. Don’t worry if you’re shocked and appalled by how unfriendly it all seems, you’re not the first. If that’s your one take-away from the kernel, that was time well spent. Now seek out friendlier communities and understand how they get stuff done. Compare them to OpenStack and ask yourself questions like “how does our reliance on voting to make decisions compare to other communities?” or “why does there seem to be less flamewars in OpenStack than elsewhere?”.

The history of open source is important, will inform how you engage with OpenStack and that, in turn, will influence how OpenStack evolves. Learn about the “free software” versus “open source” camps, and how those philosophies relate to the choice of copyleft licenses like the GPL versus permissive licenses like Apache, MIT or BSD. Are you in this for the freedom of users of your code, or are you in it to build collaborative software development communities? That contributor agreement you were asked to sign before you contributed to OpenStack – how do you feel about that?

Think about the different governance models that open-source communities adopt. Learn about benevolent dictators, project management committees, “commit bit”, consensus based decision making and the pros and cons of our representative democracy model.

Learn about the release processes various projects use. Time based versus feature based. Rapid release cycles with merge windows. Planning periods, feature freezes, release candidates, stable branches. How do different distros do this when there are so many maintainers and packages involved? We use Python a lot, how do they coordinate their release cycles?

That’s all very well, but it’s important not to be blind to the world outside open source. Understand how extreme programming and agile software development evolved. Read the Agile Manifesto. Understand how this all relates to Continuous Integration, Continuous Delivery and DevOps. We’re operating in a much different context, but is code review our variant of XP’s pair programming? Is our gated master superior to traditional post-commit CI?

You can now consider educated to a basic level. But is that enough to be an effective contributor? Do you now have everything you need to make an impact? No, far from it. The hardest part is learning to be a good human. You need to have superb communication skills, in English of course, mostly written communication skills for mailing list, gerrit and IRC discussions. We do meet twice a year in design summits, so you need to be able to present and defend your ideas in person too. You need to work on that Irish mumble of yours.

More than that, though, you need to understand people. You need to know when to be empathetic, when to be pragmatic and when you be dogmatic. When is someone’s -1 on your patch likely to be an intractable veto and when is it simply a take-it-or-leave-it suggestion? What fights are worth fighting? How can you build up kudos points by assisting your fellow contributors and when is the right time to call in some favours and spend those kudos points?

Ok, we’re ready to go! How do we put all of this into practice?

Probably the best way to start contributing to the project is by doing code reviews. You should probably be spending at least a couple of hours on code review every day. Not just because the number of code reviewers on a project has the greatest influence on its velocity, but also because its the best way to start building trust with your fellow contributors. If you can show yourself as thoughtful, committed and diligent through your code reviews, then other code reviewers will be much more inclined to prioritize your patches and less carefully scrutinize your work.

A good code reviewer manages to simultaneously focus on the little details while also considering the big picture. Try not to just leave +1 on patches, but instead a little commentary that shows the kind of things you’ve taken into consideration. Why should anyone trust that your +1 was the result of 2 hours of careful analysis, research and testing rather than just 2 minutes of coding style checking?

Also, think about who you are building up trust with. As a new code reviewer it’s probably more fruitful to provide helpful input on some meaty patches from some of the lead developers on the project. Then again, patch triage can be hugely helpful too – catch obvious problems in patches before the core reviewers ever get to the patch. Don’t forget to mentor new contributors as a code reviewer, though. Code review is the face of the project to these contributors and its your opportunity to show how you can lead by example.

Now, you obviously want to contribute code. Find some gnarly bug to fix, perhaps some race condition only rarely seen during automated tests. With all the code reviewing you’ve been doing, you’ve acquired excellent taste in coding and your work will no doubt live up to those standards. Don’t forget to write a detailed, helpful commit message and include a unit test which would catch any regression of the issue. If this is a more substantial change, you must split your change into smaller chunks where each patch represents a logical step in your progression towards the final result.

If you’re making a substantial addition like a new feature or a re-architecture, you need to document your design in some detail in a blueprint. Make sure someone reading the spec can quickly understand the problem you’re trying to solve, why it’s important and the general idea behind your solution. Then make sure there’s enough background information included that a reviewers work is made easy. Include the use cases, any relevant history, related discussions or bugs, alternative approaches considered and rejected and any security, upgrade, performance or deployer impact. Describe how your work will be tested and what documentation changes will be required.

While we’re on the subject of blueprints, don’t forget that these too need reviewers. Most projects now review the specs associated with blueprints using gerrit and so this is a way for you to demonstrate your design skills and catch things which no-one else has yet considered.

Back to code, though. Yes, it’s important to contribute to the various integrated service projects like Nova, Neutron, Swift and whatnot. However, there are a bunch of other areas where code contributions are always needed. For a start, the client projects are always forgotten. Then there’s the cross-project technical debt that the Oslo program is hard at work cleaning up. We’re also gradually porting all of OpenStack to Python 3, and this is going to be a multi year effort requiring the help of many.

We also place a huge emphasis on automated testing in OpenStack, and the awesome CI system we have doesn’t come from nowhere. You should always be ready to jump in a contribute to the infrastructure itself, tools like devstack-gate, zuul, nodepool or elastic-recheck. And, last but not least, our functional test suite, Tempest, is always desperately in need of more contributions to increase our test coverage.

Security is critical in a public-facing service like OpenStack, and there are several ways you should contribute in this area. Firstly, there is a small vulnerability management team which collaborates with each project’s -coresec team to handle privately reported security bugs, ensuring a fix is prepared for each supported branch before a coordinated, responsible disclosure of the issue first to vendors and then the wider world. Important work is this. There’s also a security group which is trying to bring together the efforts of interested parties to prepare official notices on security issues that aren’t actual vulnerabilities, develop a threat analysis process for OpenStack and maintain the OpenStack Security Guide. They need your help! Most importantly, though, you need to be security conscious as you write and review code. There’s a good chance you’ll find and report an existing vulnerability during the course of your work if you keep your eyes open!

And then there’s docs, always the poor forgotten child of any open source project. Yet OpenStack has some relatively awesome docs and a great team developing them. They can never hope to cope with the workload themselves, though, so they need you to pitch in and help perfect those docs in your area of expertise.

I mentioned bugs. We must not forget the bugs! Bugs are one way users can provide valuable contributions to the project, and we must ensure these contributions are valued so that users will continue to file bugs. With over 700 configuration options in Nova alone, the project can’t possibly test all possible combinations by itself so we rely on our users to test their own use cases and report any issues as bugs. You should help out here by setting aside some time every day to triage new bugs, making sure enough information has been provided and the bug has been appropriately tagged, categorized and prioritized.

Along those same lines, users often struggle with issues with aren’t obviously or necessarily bugs. You should also pay attention to forums like ask.openstack.org or the openstack-operators mailing list. Any outreach you can do to help users be successful with OpenStack will pay massive dividends in the long run, even just in terms of your understanding which issues are most important to real users. This outreach should extend to your attending OpenStack meetups, giving presentations on your work and listening to what users have to say.

Speaking of mailing lists, we have a hugely active openstack-dev mailing list, with over 2500 emails in April alone. This is the center of all activity happening in OpenStack at any time. You really must track what’s happening there and engage where you can help move things forward positively. It’s a struggle to keep up, but it really isn’t an option.

However, one of the side effects of openstack-dev being overloaded is that many important conversations now happen IRC. You can’t expect to be around for all of those, so make sure to remain connected and log all channels so you can catch up later.

Because conversations can be spread around multiple places, it can be helpful to link all of these conversations with little breadcrumbs. A mailing list thread might reference a gerrit review, which might reference a log of an IRC conversation, which might reference a blog post, which might reference a bug, which might reference a previous commit message which referenced a previous mailing list thread.

Don’t be fooled into thinking IRC is all about the serious stuff, though. It’s also a place where you can get to know your fellow contributors on a personal level and build up yet more of that all important trust. You will make friends working on OpenStack and some of those friendships will last longer than your involvement in OpenStack itself. That’s a hugely positive sign in any community. Beware of forming cliques, however. We need this community to be open to the most diverse set of contributors, and not all of those will buy into US-centric young white male geek humour, for example.

Speaking of cliques, it’s popular to accuse OpenStack developers on being so self-absorbed that the needs of real operators and users are ignored. That OpenStack developers aren’t held responsible for the real world consequences of the decisions they make. “You write code differently when you carry a pager”. Lorin Hochstein proposed an “Adopt a Dev” program where operators could invite individual developers to shadow them for a few days and share their experience in the terms of a summary, bug reports and blueprints. Basically, you should take any opportunity you can to get your hands dirty and help operate a production OpenStack service.

Related to the needs of operators are the deployment, configuration and operational tools out there which desperately need contributions with people more familiar with the dirty details of how the software works. Many developers use devstack to deploy their development clouds, but there’s huge benefit in occasionally deploying something more production-like and contributing to whatever tool you used. TripleO is a great deployment effort to contribute to because it’s attempting to create a space where everyone interested in deployment can collaborate, but also because it closely tracks the development branch of OpenStack.

Once you have succeeded at making an impact as an individual contributor, you should look to extend your leadership efforts beyond simply leading by example. Naturally, you’ll tend towards volunteering for the responsibility of the PTL position on whichever program you contribute most to. To demonstrate your willingness and trustworthiness for the position, perhaps you’ll suggest the PTL delegate some of their responsibilities to you.

Your leadership interests should extend beyond a single project too. In some ways, the kind of cross-project issues considered by the Technical Committee are as important as the per-project responsibilities of PTLs. Do you have strong opinions on how, why and when should add new programs or Integrated projects. If not, why not?

The governance of OpenStack and the shared responsibility for the future direction of OpenStack extends beyond the TC and PTL’s governance of the project itself, to the role of the Foundation Board of Directors in protecting, empowering and promoting the project as well as ensuring there’s a healthy commercial and non-commercial ecosystem around the project. Do you care how the TC and board divide their responsibilities? Or how much explicit corporate influence is appropriate in the technical decision making of the project? Or how the board makes legal decisions which directly impact the project? Or how individual members elect their representatives on the board? You should.

Wait, wait, I’m forgetting a bunch of stuff. You should care deeply about bringing contributors on board and participate in the awesome OPW and GSoC programs. It’s important to keep track of how the project is perceived, so you should read any articles published about the project and follow even our worst detractors on twitter. Watch carefully how our major competitors like AWS and GCE are evolving. Make sure to keep on relevant new developments like NFV or Docker. Keep an eye on new projects on Stackforge to track how they develop.

Huh, wait. You’re probably employed to work full time on the project, right? Well, you really need to learn how to wear upstream and downstream “hats”. You need to understand how you can help your employer be successful with their objectives around the project. You need to be able to reconcile any apparent conflicts between your employers’ needs and the best interests of the project. This is not a zero sum game. Meet with your employer’s customers and partners, help deliver what OpenStack product or service your employer is providing, mentor colleagues on how to successfully engage with the project and be the bridge over the upstream and downstream gap.

Above all, through all of this, be nice to everyone you encounter and wear a smile.

BZZZT … BURNOUT ALERT

I’m obviously being facetious, right? There’s no way anyone can possibly live up to those expectations and live to tell the tale?

It’s pretty obvious when you put it all together like this that these are unreasonable expectations. The hero of this tale does not exist. Many of us have tried to be this person, but it’s just not possible. Read into this, if you like, a very personal tale of burnout caused by unreasonable self-imposed expectations.

But really, what I want to get across today is that you don’t need to be this hero in order to contribute. Far from being too many active monthly contributors, five hundred is just the tip of the iceberg. Why shouldn’t every attendee of every OpenStack meetup be able to contribute in some small way?

When mentoring new Red Hat engineers, my basic advice is always “find your niche”. Find something that takes your interest and that you can see an obvious path towards making a significant impact, and go deep! Ignore pretty much everything else and do your thing. Maybe after a while you’ll have got the ball rolling of its own accord and that there are other areas you can now make an equally big impact on. Or perhaps you’ll stick with this niche and continue to make an impact doing it over the longer term.

One of my favorite examples of a less likely niche is bug triage. Back in the summer of 2001 when I started seriously contributing the GNOME project and became a maintainer of its CORBA ORB, ORBit, another new contributor to the project called Luis Villa posted this email:

Hey, everybody. By way of introduction: I’m the new bugmaster at Ximian. As some of you may have noticed, I’m slowly moving towards cleaning out evo and RC bugs from bugzilla.gnome and into bugzilla.ximian.

Luis went on to breath new life into GNOME’s “bugsquad”, helped put in place a highly effective bug triage process and taught the GNOME community how to truly value and celebrate the contributions of both bug reporters and bug triagers. If you want to make fame and fortune in the open source world, how many people would pick bug triage as the place to start? Well, Luis did and made a huge impact, before moving on to engineering management and then giving it all up to go to law school. He is now Assistant General Counsel for the Wikimedia Foundation.

There’s a real “find your niche” lesson in that story, but also a lesson that we as a community need to learn to truly value and celebrate all of the myriad of different ways that contributors can help the project. Rather than judge others based on how they’re not contributing, rather than feel exasperated when so few others share your passion for a particular niche no matter how important it seems to you personally, we as a community need to acquire a greater level of empathy for our fellow contributors.

We also need to experiment with ways of running the project so that different roles and niches are appropriately recognized. Does the focus we put on PTLs detract from the valuable project management contributions others make? Are official programs the only way of recognizing the importance of particular areas? If programs are the only way, do we need to be more open to creating programs wherever a group of people have coalesced around some particular effort? Do we need to explicitly raise the profiles of those contributors doing hard behind-the-scenes work in areas that we don’t typically recognize? Are we building a culture that places too much emphasis on recognition and instead roll back some of the ways we recognize people now?

Lot’s of questions, few answers. But hopefully this can get the conversation started.

tl;dr – the affiliation limit means that at most one of the two Red Hat affiliated candidates can be elected. The cumulative voting system makes it likely that both of us running seriously damages both of our chances of being elected. A preferential voting system like Condorcet or STV would not have this problem.

—

At Red Hat, those of us who contribute to OpenStack take very seriously our responsibility to put what’s good for the project first and foremost in our minds – to wear our “upstream hat”, as we like to say. That’s especially true for me and Russell Bryant.

However, now that the candidate list for the 2014 OpenStack Individual Member Director Elections has been finalized, we find ourselves wrestling with the fact that Russell and I are both running as candidates. Two aspects of our election system make this a problem. First, the cumulative voting system means that those who would be happy to vote for either me or Russell are forced to choose between us – essentially, we are damaging each others’ chances of being elected. Secondly, the affiliation limit means that even if we were both lucky enough to receive enough votes to be elected, one of us would be eliminated by the limit.

The combination of these two issues means that we have to factor our affiliation into our decision. The rules place affiliation front and centre in election system, even though Individual Member Directors are not elected to represent their employer.

Now, I’m personally guilty of not pushing this election system issue hard enough over this past year. At one point I favoured experimenting with a tweak to the cumulative system over a more dramatic change because I found the prospect of getting a majority of over 25% of our enormous electorate to vote in favour of a change so daunting. I want to be completely open about this decision we now face because I want to help raise awareness about how important an issue it is.

Given that Red Hat is a Platinum Member and has a automatic seat on the board, the options we’re weighing up are:

1. Continue with both Russell and me on the ballot, accepting the risk that we’re damaging each others chances.
2. I or Russell remove ourselves from the ballot, giving the other of us the best possible chance of being elected.
3. Brian Stevens steps down from the board and I or Russell takes his place, giving whichever of us remains on the ballot the best possible chance of being elected.

It’s not an easy decision. We both feel we have something to offer on the board. Both of us would be very proud to be elected to represent the Individual Members. Both of us feel that Brian Stevens (our CTO who we greatly respect) is the best possible representative for Red Hat on the board.

We will make a decision on this before the election, but right now we don’t see any of the options as being particularly better than the others. But, at the very least, I hope everyone will find this useful as a concrete example of why our election system needs to change.

I’ve been following the “what is core?” conversation since the “Incubation Update” committee completed its work some time ago. I’m really happy to see Rob, Alan and others put so much work into moving this along, but each time I try to catch up on progress I find myself a little bewildered.

Since it’s going to be a big topic during next week’s board meeting, I figured I’d try to get my thoughts together here.

What’s it all About?

Even the question bothers me – "what is core?". Why is that an important question?

One reason for its importance is that the bylaws say:

The Core OpenStack Project means the software modules […] for which an OpenStack trademark may be used

In other words, Heat can't call itself "OpenStack Orchestration" unless the Board of Directors approve Heat as part of "The Core OpenStack Project". If that was all this was about, I'd be like "hell yes! Heat should be known as OpenStack Orchestration!". But, it's clearly not just about this, so I tend to ignore this aspect. I don’t actually think much of what is being discussed is all that relevant to this particular issue.

So, what else is this all about? Well, one clue is the emphasis on testing in Rob’s list of “10 core positions”.

OpenStack Core means passing all “must-pass” tests

This is an example of where I get really confused. Maybe I’m just getting hung up on language. I think we’re talking about cloud providers, distributions, vendors, deployers, etc. self-certifying themselves using a test suite in order that they can use the OpenStack trademark. But are we talking about labelling these self-certified clouds as “Core”? Aren’t we making this all very confusing by using the term “Core” both to describe the self-certified clouds/products and also to describe the subset of OpenStack they are required to include? I’d phrase this as:

The OpenStack Core definition includes a list of tests which certified clouds must pass

Anyway, I’m not going to nitpick my way through all of this. I think I understand the intent here, but I like to approach this from an entirely different angle.

A Market of Interoperable OpenStack Providers

(Yes, that is Simon Wardley’s terminology)

We all know that one of the basic goals of OpenStack is for there to be a bunch of public clouds available to users around the world and that interoperability between these public OpenStack clouds will make it easier for users to switch cloud providers and encourage competition between these providers.

To my mind, that’s what this whole “what is core?” conversation is really about. We want to use an OpenStack trademark program to help build this marketplace by enforcing interoperability between clouds which use the OpenStack trademark.

(And yes, I understand that the “what is core?” discussion is also related to questions like what is required of an OpenStack distribution. I’m focusing on the question of public cloud interoperability because I think it’s the most important. IMHO, the whole conversation is pointless unless it moves this particular topic along quickly.)

To do that, we need to define which APIs these clouds must expose, or as Rob puts it – which use cases these clouds must support. To take a simple example, should these clouds be required to expose the Glance API for uploading images?

And here’s where I get confused again. Why are we talking about “what is core?” when we could simply say “which APIs are required to be exposed by certified OpenStack clouds?”. Again, I’m trying not to be a pedant, but the way the question is framed leaves me unsure whether we’re really all talking about the same thing.

I think the focus should be on immediate baby-steps towards kick-starting this marketplace. One simple question – if and when we certify the first batch of interoperable clouds, would we rather have a smaller number of big clouds included or a large number of smaller clouds? In terms of resource capacity provided by this marketplace, I guess it’s more-or-less the same thing.

Let’s assume we absolutely want (at least) a small number of the bigger providers included in the program at launch. Let’s say “small number” equals “Rackspace and HP”. And let’s assume both of these providers are very keen to help get this program started. Isn’t the obvious next baby-step to get representatives of those two providers to figure out exactly what level of interoperability they already have and also what improvements to that they can make in the short term?

If we had that report, we could next do a quick “sniff test” comparing this to many of the other OpenStack clouds out there to make sure we haven’t just picked two clouds with an unusual set of compatible APIs. Then the board could make a call on whether this represents a reasonable starting point for the requirements of OpenStack clouds.

No, this isn’t perfect. But it would be a genuine step forward towards being able to certify some clouds. We would have some data on commonalities and have made a policy decision on what commonalities are required. It would be a starting point.

OpenStack Compatible?

The next big decision for the board is whether a cloud which uses the OpenStack trademark must actually be a deployment of OpenStack’s code. Is this a question of “OpenStack clouds” or “OpenStack compatible clouds”?

I do think it would be damaging to OpenStack if this marketplace took off and was dominated by providers which don’t use or contribute to OpenStack. But, as Rob says, the trademark isn’t the only way of avoiding this situation – there’s also our “velocity and culture”.

I struggle to see how trademark requirements around the use of OpenStack code would work. How do you define which code must be used? Must that code be used unmodified? If not, how much can you change? What does it even mean to “use” a piece of code? That user requests must be executed using that code? Maybe this would just be a “yes, we use and contribute to OpenStack” good faith statement which the Foundation would make a judgment call on? If so, how do we ensure transparency and fairness around how that call is made?

If this question proves to be a stumbling block, I’d prefer to see an “OpenStack compatible cloud” trademark program established quickly. Getting these interoperability guarantees in place for our users takes priority over ensuring that certified providers actually use and contribute to OpenStack.

Conclusion of Sorts

I think our immediate concern should be kicking off a trademark program for certifying interoperability between OpenStack clouds. I’m frustrated whenever I think this “what is core?” discussion is tackling questions which aren’t immediate blockers to making progress on this program.

The board has to make (or oversee the making of) some policy questions.

Firstly, what APIs are required in OpenStack™ clouds? I’d favour starting to answer this by first looking at the current interoperability levels between existing clouds.

Secondly, whether and how we insist on the use of OpenStack code in OpenStack™ clouds. If this turns out to be a difficult problem, then I think we should just start with an “OpenStack™ compatible cloud” program.