Archive for the ‘openstack’ Category

An Ideal OpenStack Developer

Friday, June 6th, 2014

(This is a prose version of a talk I gave at OpenStack meetups in Israel and London recently. Apologies for the wordiness.)

In a recent update Jonathan gave to the Board of Directors, we described how OpenStack has had 2,130 contributors to date and 466 of those are active on a monthly basis. That’s an incredible statistic. There’s no doubt OpenStack has managed to attract an unusual number of contributors and, for such a complex project, made it relatively easy for them to contribute.

However, this isn’t just a numbers game. I often hear mutterings that a much smaller, focused group could achieve the same velocity that OpenStack is achieving. In some sense that’s true, but I think that the diversity of interests and priorities is the energy that a community like OpenStack thrives on.

The question then is how to improve the overall quality of our large number of contributors. In order to do that, we need to be able to set expectations. What do we expect and value from our contributors?

What I’m going to attempt to do here is define The Prototypical OpenStack Developer. The ideal that we should aspire to. The standard that all contributors should be held to.

(But … bear with me here. I’m being a little tongue-and-cheek.)

Ok. Where do we start? How do we begin to forge this hero from the raw resources we are presented with?

Let’s start with the basics. The breadth and depth of knowledge you need on a variety of computing topics.

On virtualization, you could start with KVM. You should know about CPU extensions such as Intel’s VT-x and I/O virtualization with VT-d and PCI SR-IOV. Some knowledge of the history of software based virtualization and paravirtualization would be nice context too. Now understand the responsibilities of the KVM kernel module versus the userspace component, qemu. How does qemu emulate various devices? How does live migration work? How does a hypervisor use page table flags to track dirty pages during a migration?

And there’s probably little point in understanding all of this without understanding the x86 architecture in some detail. Understanding how it compares to RISC architectures would be no harm. Memory segmentation, MMUs, page tables are all fun topics. You really can’t get into this without learning a bit of assembly, at least the basic idea. The history of x86, from real/protected mode to modern day PAE or x86-64 are all important to understand. Ignore Itanium, though. It’s not enough to just understand the CPU, though, you need to go beyond and think about how that CPU interacts with peripherals using DMA and buses like PCI.

And, honestly, if you go this far you may as well understand basic digital systems theory, like how you can construct a counter or register from a set of basic logic gates …

Woah, I think I’ve digressed a little. That’s virtualization. Do the same for storage and networking. I’ll leave that as an exercise for the reader.

That’s just the concept behind the basic resources managed by OpenStack, though. It’s a pretty complicated distributed system, so it’s pretty essential you do some reading on that topic. What do terms like “quorum” and “consensus” mean? Why do people describe the Paxos algorithm as “quicksort of distributed systems”? What do people mean when they describe OpenStack as a “shared nothing” architecture, and are they crazy? How would you describe OpenStack’s approach to fault tolerance?

And obviously related to all of this is the need for deep knowledge of databases and messaging systems. We seem to have a large number of ex-MySQL consultants on this project, but don’t let that be an excuse. You know what foreign keys and cross-table joins are, right? And you really need to know the kind of operations which will simply lock individual rows rather than an entire table. For messaging, there’s a little research you can do there. We’re all about AMQP in OpenStack, but there’s been a few other messaging protocols in the past. My personal favorite is CORBA. What’s the difference between a broker, router and peer-to-peer based architecture? What’s this “fanout” and “topic” things we talk about in messaging? Incidentally, you know that we’re not actually using the standard AMQP protocol in OpenStack, right?

You needn’t have touched a line of code at this point. But, if you’re going to contribute to OpenStack, you need to code, right? Almost certainly in Python, but we like ourselves a little Bash too. With Python, it’s important to understand not just the syntax from the most basic to the more advanced topics like iterators, decorators, context managers and metaclasses. You also need to have a good knowledge of the huge number of python libraries out there, inside and outside the core Python distribution. We need true Pythonistas. Oh, and we’re in the process of porting to Python 3, so make sure you understand the differences between Python 2 and 3.

But wait, wait. That’s no good. You can’t just dive straight into Python. You need to start with C. Allocate and free your own memory, damnit. You can’t go through life without learning about pointers. Now learn how to use threads and the various synchronization primitives out there, and why it’s all just terrible. Now learn about asynchronous I/O techniques; what an event loop is, how you use select() and non-blocking sockets to write a single-threaded server which processes requests from multiple clients. Oh, Richard Stevens. My hero. Don’t be afraid to read a few RFCs.

Speaking of authors, we forgot algorithms. Yes, those. Just carefully study all three volumes of Knuth.

Now, before returning to Python, perhaps you should implement a REST API in Java using JAX-RS and a web UI using Ruby on Rails. Hang out with the cool kids and port your UI to Sinatra, before realizing that’s not cool anymore and switching to Node.js.

You might be ready to contribute some code to OpenStack at this point. But, I hate to think of anyone writing software without having a full appreciation of the user experience design we’re driving towards. We don’t want the inmates running the asylum, do we? Which “personas” are we designing for? “As a web developer, I want to launch a virtual machine to test my code in.”

Wait, we forgot tools. You can’t get anything done without knowing your tools. You’re going to do all of your work on Linux, whether that be in VMs or by running Linux on your main machine. If you’re a serious person, you need to learn emacs. You’re going to become very close friends with grep and sed, so learn yourself regular expressions. Lazy and greedy regexs, both. You know how to do a HTTP POST with curl, right?

Ah, git! Oh, the glorious git! You can never learn too much about git. It’s the gift that keeps on giving. If you think I’m joking, spend some time getting to know interactive rebasing. Reordering, editing, squashing and splitting commits! Re-writable history! Where have you been all my life? No git detail is too obscure to ignore. Learn how a tilde is different from a caret in revision parameters. How you can delete branches by leaving out the first part of a refspec in a git-push. Force override, exciting! Is your mind blown yet? No? Find out how git’s reflog is a history of history!

(Give me a second to calm down, here)

Now, you’ve got to realize something. Based on everything you’ve learned so far, you could probably write OpenStack on your own. But that’s not what’s going on here. You’re collaborating. You’re following a process. How we collaborate and why we follow certain processes is a more complex, involved and undocumented topic than anything you’ve learned so far.

To really understand how we get stuff done in OpenStack, you need to be steeped in open source culture. Understand what we mean when we say things like “rough consensus and running code” or “do-acry”.

Perhaps start by following the linux-kernel mailing list for a few months, watching how controversial discussions are worked through and the subtleties that determine who holds the balance of power and influence. Don’t worry if you’re shocked and appalled by how unfriendly it all seems, you’re not the first. If that’s your one take-away from the kernel, that was time well spent. Now seek out friendlier communities and understand how they get stuff done. Compare them to OpenStack and ask yourself questions like “how does our reliance on voting to make decisions compare to other communities?” or “why does there seem to be less flamewars in OpenStack than elsewhere?”.

The history of open source is important, will inform how you engage with OpenStack and that, in turn, will influence how OpenStack evolves. Learn about the “free software” versus “open source” camps, and how those philosophies relate to the choice of copyleft licenses like the GPL versus permissive licenses like Apache, MIT or BSD. Are you in this for the freedom of users of your code, or are you in it to build collaborative software development communities? That contributor agreement you were asked to sign before you contributed to OpenStack – how do you feel about that?

Think about the different governance models that open-source communities adopt. Learn about benevolent dictators, project management committees, “commit bit”, consensus based decision making and the pros and cons of our representative democracy model.

Learn about the release processes various projects use. Time based versus feature based. Rapid release cycles with merge windows. Planning periods, feature freezes, release candidates, stable branches. How do different distros do this when there are so many maintainers and packages involved? We use Python a lot, how do they coordinate their release cycles?

That’s all very well, but it’s important not to be blind to the world outside open source. Understand how extreme programming and agile software development evolved. Read the Agile Manifesto. Understand how this all relates to Continuous Integration, Continuous Delivery and DevOps. We’re operating in a much different context, but is code review our variant of XP’s pair programming? Is our gated master superior to traditional post-commit CI?

You can now consider educated to a basic level. But is that enough to be an effective contributor? Do you now have everything you need to make an impact? No, far from it. The hardest part is learning to be a good human. You need to have superb communication skills, in English of course, mostly written communication skills for mailing list, gerrit and IRC discussions. We do meet twice a year in design summits, so you need to be able to present and defend your ideas in person too. You need to work on that Irish mumble of yours.

More than that, though, you need to understand people. You need to know when to be empathetic, when to be pragmatic and when you be dogmatic. When is someone’s -1 on your patch likely to be an intractable veto and when is it simply a take-it-or-leave-it suggestion? What fights are worth fighting? How can you build up kudos points by assisting your fellow contributors and when is the right time to call in some favours and spend those kudos points?

Ok, we’re ready to go! How do we put all of this into practice?

Probably the best way to start contributing to the project is by doing code reviews. You should probably be spending at least a couple of hours on code review every day. Not just because the number of code reviewers on a project has the greatest influence on its velocity, but also because its the best way to start building trust with your fellow contributors. If you can show yourself as thoughtful, committed and diligent through your code reviews, then other code reviewers will be much more inclined to prioritize your patches and less carefully scrutinize your work.

A good code reviewer manages to simultaneously focus on the little details while also considering the big picture. Try not to just leave +1 on patches, but instead a little commentary that shows the kind of things you’ve taken into consideration. Why should anyone trust that your +1 was the result of 2 hours of careful analysis, research and testing rather than just 2 minutes of coding style checking?

Also, think about who you are building up trust with. As a new code reviewer it’s probably more fruitful to provide helpful input on some meaty patches from some of the lead developers on the project. Then again, patch triage can be hugely helpful too – catch obvious problems in patches before the core reviewers ever get to the patch. Don’t forget to mentor new contributors as a code reviewer, though. Code review is the face of the project to these contributors and its your opportunity to show how you can lead by example.

Now, you obviously want to contribute code. Find some gnarly bug to fix, perhaps some race condition only rarely seen during automated tests. With all the code reviewing you’ve been doing, you’ve acquired excellent taste in coding and your work will no doubt live up to those standards. Don’t forget to write a detailed, helpful commit message and include a unit test which would catch any regression of the issue. If this is a more substantial change, you must split your change into smaller chunks where each patch represents a logical step in your progression towards the final result.

If you’re making a substantial addition like a new feature or a re-architecture, you need to document your design in some detail in a blueprint. Make sure someone reading the spec can quickly understand the problem you’re trying to solve, why it’s important and the general idea behind your solution. Then make sure there’s enough background information included that a reviewers work is made easy. Include the use cases, any relevant history, related discussions or bugs, alternative approaches considered and rejected and any security, upgrade, performance or deployer impact. Describe how your work will be tested and what documentation changes will be required.

While we’re on the subject of blueprints, don’t forget that these too need reviewers. Most projects now review the specs associated with blueprints using gerrit and so this is a way for you to demonstrate your design skills and catch things which no-one else has yet considered.

Back to code, though. Yes, it’s important to contribute to the various integrated service projects like Nova, Neutron, Swift and whatnot. However, there are a bunch of other areas where code contributions are always needed. For a start, the client projects are always forgotten. Then there’s the cross-project technical debt that the Oslo program is hard at work cleaning up. We’re also gradually porting all of OpenStack to Python 3, and this is going to be a multi year effort requiring the help of many.

We also place a huge emphasis on automated testing in OpenStack, and the awesome CI system we have doesn’t come from nowhere. You should always be ready to jump in a contribute to the infrastructure itself, tools like devstack-gate, zuul, nodepool or elastic-recheck. And, last but not least, our functional test suite, Tempest, is always desperately in need of more contributions to increase our test coverage.

Security is critical in a public-facing service like OpenStack, and there are several ways you should contribute in this area. Firstly, there is a small vulnerability management team which collaborates with each project’s -coresec team to handle privately reported security bugs, ensuring a fix is prepared for each supported branch before a coordinated, responsible disclosure of the issue first to vendors and then the wider world. Important work is this. There’s also a security group which is trying to bring together the efforts of interested parties to prepare official notices on security issues that aren’t actual vulnerabilities, develop a threat analysis process for OpenStack and maintain the OpenStack Security Guide. They need your help! Most importantly, though, you need to be security conscious as you write and review code. There’s a good chance you’ll find and report an existing vulnerability during the course of your work if you keep your eyes open!

And then there’s docs, always the poor forgotten child of any open source project. Yet OpenStack has some relatively awesome docs and a great team developing them. They can never hope to cope with the workload themselves, though, so they need you to pitch in and help perfect those docs in your area of expertise.

I mentioned bugs. We must not forget the bugs! Bugs are one way users can provide valuable contributions to the project, and we must ensure these contributions are valued so that users will continue to file bugs. With over 700 configuration options in Nova alone, the project can’t possibly test all possible combinations by itself so we rely on our users to test their own use cases and report any issues as bugs. You should help out here by setting aside some time every day to triage new bugs, making sure enough information has been provided and the bug has been appropriately tagged, categorized and prioritized.

Along those same lines, users often struggle with issues with aren’t obviously or necessarily bugs. You should also pay attention to forums like ask.openstack.org or the openstack-operators mailing list. Any outreach you can do to help users be successful with OpenStack will pay massive dividends in the long run, even just in terms of your understanding which issues are most important to real users. This outreach should extend to your attending OpenStack meetups, giving presentations on your work and listening to what users have to say.

Speaking of mailing lists, we have a hugely active openstack-dev mailing list, with over 2500 emails in April alone. This is the center of all activity happening in OpenStack at any time. You really must track what’s happening there and engage where you can help move things forward positively. It’s a struggle to keep up, but it really isn’t an option.

However, one of the side effects of openstack-dev being overloaded is that many important conversations now happen IRC. You can’t expect to be around for all of those, so make sure to remain connected and log all channels so you can catch up later.

Because conversations can be spread around multiple places, it can be helpful to link all of these conversations with little breadcrumbs. A mailing list thread might reference a gerrit review, which might reference a log of an IRC conversation, which might reference a blog post, which might reference a bug, which might reference a previous commit message which referenced a previous mailing list thread.

Don’t be fooled into thinking IRC is all about the serious stuff, though. It’s also a place where you can get to know your fellow contributors on a personal level and build up yet more of that all important trust. You will make friends working on OpenStack and some of those friendships will last longer than your involvement in OpenStack itself. That’s a hugely positive sign in any community. Beware of forming cliques, however. We need this community to be open to the most diverse set of contributors, and not all of those will buy into US-centric young white male geek humour, for example.

Speaking of cliques, it’s popular to accuse OpenStack developers on being so self-absorbed that the needs of real operators and users are ignored. That OpenStack developers aren’t held responsible for the real world consequences of the decisions they make. “You write code differently when you carry a pager”. Lorin Hochstein proposed an “Adopt a Dev” program where operators could invite individual developers to shadow them for a few days and share their experience in the terms of a summary, bug reports and blueprints. Basically, you should take any opportunity you can to get your hands dirty and help operate a production OpenStack service.

Related to the needs of operators are the deployment, configuration and operational tools out there which desperately need contributions with people more familiar with the dirty details of how the software works. Many developers use devstack to deploy their development clouds, but there’s huge benefit in occasionally deploying something more production-like and contributing to whatever tool you used. TripleO is a great deployment effort to contribute to because it’s attempting to create a space where everyone interested in deployment can collaborate, but also because it closely tracks the development branch of OpenStack.

Once you have succeeded at making an impact as an individual contributor, you should look to extend your leadership efforts beyond simply leading by example. Naturally, you’ll tend towards volunteering for the responsibility of the PTL position on whichever program you contribute most to. To demonstrate your willingness and trustworthiness for the position, perhaps you’ll suggest the PTL delegate some of their responsibilities to you.

Your leadership interests should extend beyond a single project too. In some ways, the kind of cross-project issues considered by the Technical Committee are as important as the per-project responsibilities of PTLs. Do you have strong opinions on how, why and when should add new programs or Integrated projects. If not, why not?

The governance of OpenStack and the shared responsibility for the future direction of OpenStack extends beyond the TC and PTL’s governance of the project itself, to the role of the Foundation Board of Directors in protecting, empowering and promoting the project as well as ensuring there’s a healthy commercial and non-commercial ecosystem around the project. Do you care how the TC and board divide their responsibilities? Or how much explicit corporate influence is appropriate in the technical decision making of the project? Or how the board makes legal decisions which directly impact the project? Or how individual members elect their representatives on the board? You should.

Wait, wait, I’m forgetting a bunch of stuff. You should care deeply about bringing contributors on board and participate in the awesome OPW and GSoC programs. It’s important to keep track of how the project is perceived, so you should read any articles published about the project and follow even our worst detractors on twitter. Watch carefully how our major competitors like AWS and GCE are evolving. Make sure to keep on relevant new developments like NFV or Docker. Keep an eye on new projects on Stackforge to track how they develop.

Huh, wait. You’re probably employed to work full time on the project, right? Well, you really need to learn how to wear upstream and downstream “hats”. You need to understand how you can help your employer be successful with their objectives around the project. You need to be able to reconcile any apparent conflicts between your employers’ needs and the best interests of the project. This is not a zero sum game. Meet with your employer’s customers and partners, help deliver what OpenStack product or service your employer is providing, mentor colleagues on how to successfully engage with the project and be the bridge over the upstream and downstream gap.

Above all, through all of this, be nice to everyone you encounter and wear a smile.

BZZZT … BURNOUT ALERT

I’m obviously being facetious, right? There’s no way anyone can possibly live up to those expectations and live to tell the tale?

It’s pretty obvious when you put it all together like this that these are unreasonable expectations. The hero of this tale does not exist. Many of us have tried to be this person, but it’s just not possible. Read into this, if you like, a very personal tale of burnout caused by unreasonable self-imposed expectations.

But really, what I want to get across today is that you don’t need to be this hero in order to contribute. Far from being too many active monthly contributors, five hundred is just the tip of the iceberg. Why shouldn’t every attendee of every OpenStack meetup be able to contribute in some small way?

When mentoring new Red Hat engineers, my basic advice is always “find your niche”. Find something that takes your interest and that you can see an obvious path towards making a significant impact, and go deep! Ignore pretty much everything else and do your thing. Maybe after a while you’ll have got the ball rolling of its own accord and that there are other areas you can now make an equally big impact on. Or perhaps you’ll stick with this niche and continue to make an impact doing it over the longer term.

One of my favorite examples of a less likely niche is bug triage. Back in the summer of 2001 when I started seriously contributing the GNOME project and became a maintainer of its CORBA ORB, ORBit, another new contributor to the project called Luis Villa posted this email:

Hey, everybody. By way of introduction: I’m the new bugmaster at Ximian. As some of you may have noticed, I’m slowly moving towards cleaning out evo and RC bugs from bugzilla.gnome and into bugzilla.ximian.

Luis went on to breath new life into GNOME’s “bugsquad”, helped put in place a highly effective bug triage process and taught the GNOME community how to truly value and celebrate the contributions of both bug reporters and bug triagers. If you want to make fame and fortune in the open source world, how many people would pick bug triage as the place to start? Well, Luis did and made a huge impact, before moving on to engineering management and then giving it all up to go to law school. He is now Assistant General Counsel for the Wikimedia Foundation.

There’s a real “find your niche” lesson in that story, but also a lesson that we as a community need to learn to truly value and celebrate all of the myriad of different ways that contributors can help the project. Rather than judge others based on how they’re not contributing, rather than feel exasperated when so few others share your passion for a particular niche no matter how important it seems to you personally, we as a community need to acquire a greater level of empathy for our fellow contributors.

We also need to experiment with ways of running the project so that different roles and niches are appropriately recognized. Does the focus we put on PTLs detract from the valuable project management contributions others make? Are official programs the only way of recognizing the importance of particular areas? If programs are the only way, do we need to be more open to creating programs wherever a group of people have coalesced around some particular effort? Do we need to explicitly raise the profiles of those contributors doing hard behind-the-scenes work in areas that we don’t typically recognize? Are we building a culture that places too much emphasis on recognition and instead roll back some of the ways we recognize people now?

Lot’s of questions, few answers. But hopefully this can get the conversation started.

May 11 OpenStack Foundation Board Meeting

Saturday, May 17th, 2014

The OpenStack Foundation Board of Directors met in-person in advance of the OpenStack Summit in Atlanta. This my informal recollection of the meeting. It’s not an official record, etc.

Unlike previous meetings held in advance of summits, this meeting only ran from 09:00 to 14.30 at which time we switched venue for the first ever joint board of directors and technical committee meeting.

I’m about to head off on vacation for a week, so I figured I’d do my best to briefly cover some of the topics covered during the meeting.

Jonathan’s Update

After the usual preliminaries, we began the meeting with Jonathan Bryce (in his role as Executive Director) giving the board an update from the Foundation’s perspective.

One of the more interesting slides in Jonathan’s updates is always the latest statistics showing community and ecosystem growth. We now have over 355 companies supporting the foundation, over two thousand total contributors and almost five hundred active contributors every month. Over 17,000 commits were merged during the Icehouse release cycle, an increase of 25% from Havana. This level growth is just phenomenal.

Jonathan also talked about the growth in visitors to the openstack.org website and made some interesting observations about the geographical spread of the visitors. The top 4 countries seen in the stats are the U.S., India, China and France. That France figures so highly in the stats is a good sign in advance of the summit in Paris in November.

Next, Jonathan moved on to talk about the week ahead in Atlanta. Once again, we’re seeing a huge increase in the level of interest in the event with over 4,500 attendees compared to the roughly 3,000 attendees in Hong Kong. Running an even of this size is a massive undertaking and Jonathan mentioned one crazy statistic – the foundation had over 23,000 pieces printed for the event and had to spread those orders over three printing companies in order to be able to do it.

A big emphasis for the week was an increased focus on users and operators. And, interestingly, there were roughly 800 developers and 700 operators signed up for the event. All were agreed that it’s a very healthy sign to see so many operators attend.

One comment from Jonathan triggered some debate – that the event was turning into a broader cloud industry event rather than strictly limited to just OpenStack. Some board members raised a concern that the event shouldn’t become completely generic and the focus should always be on OpenStack and its ecosystem. Jonathan clarified that this is the intent.

Jonathan also talked about the geographical diversity of attendees at the summit. People were coming from over 55 countries, but 81% attendees were from the U.S. In contrast, in Hong Kong, the percentage of US attendees were more like 40% and Jonathan felt that this showed the importance of regularly holding summits outside of the U.S.

Finances

Jonathan also walked the board through and update on the foundation’s financial position. Operating income was 3% above their predictions and expenses was down 7%. This has left the foundation with $7.8M in the bank, as part of Jonathan’s goal to build up a substantial war-chest to ensure the foundation’s stability even in the event of unforseeable events.

The summit in Atlanta was predicted to make a loss of $50k but was on track to make a profit. And yet, while it was predicted to be a $2.7M event, it was turning out to be a $4M event. The situation will be very different in Paris because of different cost structures and the event is expected to make a loss. While on the topic, some board members requested that the board be more closely involved in choosing the location of future summits. Jonathan was happy to facilitate that and expected to be able to give the board an update in July.

Jonathan next gave a detailed update on the foundation’s application for US federal tax exempt status. He explained that while we are Delaware incorporated, non-stock, non-profit foundation we have not yet been granted 501(c)(6) status by the IRS. After providing the IRS with additional information in November, the IRS returned an initial denial in March and the foundation filed a protest in April.

The objections from the IRS boil down to their feeling (a) that the foundation is producing software and, as such, is “carrying on a normal line of business”, (b) that the foundation isn’t improving conditions for the entire industry and (c) that the foundation is performing services for its members. Jonathan explained why the foundation feels those objections aren’t warranted and that the OpenStack foundation is fundamentally no different from other similar 501(c)(6) organizations like the Linux Foundation. He explained that other similar organizations were going through similar difficulties and he feels it is incumbent on the foundation to continue to challenge this in order to avoid a precedent being set for other organizations in the future. Overall, Jonathan seemed confident about our position while also feeling that the outcome is hard to predict with complete certainty. This conversation continued for some time and, because of the interest, the board moved to establish a committee to track the issue consisting of the existing members of the finance committee along with Eileen, Todd and Sean.

Trademark Framework

Jonathan moved on to give an update on some changes the foundation have made around the trademark programs in place for commercial uses of the mark. The six logos previously used were causing too much confusion so the foundation has merged these into “Powered By OpenStack” and “OpenStack Compatible” marks.

There followed some debate and clarifications were given, before some members expressed concern that the board had not been adequately consulted on the change. That objection seemed unwarranted to me given that Jonathan had briefed the board on the change in advance of implementing it.

Staying on topic of trademark programs, Boris took the floor and gave an update on the DriverLog work his team has been working on. He request the board use the output of DriverLog to enforce quality standards for the use of the OpenStack compatible mark in conjunction with Nova, Neutron and Cinder drivers. There was rather heated debate on the implications of this, particularly around whether drivers would be required to be open-source and/or merged in trunk.

Several board members objected to the fact that this proposal wasn’t on the agenda and the board hadn’t been provided with supporting materials in advance of the meeting. Boris committed to providing said material to the board before revisiting the issue.

Defcore

Next up, Rob and Josh gave an update on the progress of their DefCore initiative. Rather than attempt to repeat the background here, it’s probably best to read Rob’s own words.

Once the background was covered, the board spent some time considering the capabilities scoring matrix where each capability (concretely, capabilities are groups of Tempest tests) is scored against 12 selection criteria. This allows the capabilities to be ranked so that the board can make an objective judgment on which capabilities should be considered “must have”. There appeared to be generally good consensus around the approach, but a suggestion was made to consider more graduated scoring of the criteria (e.g. 1-5 rather than 0 or 1).

The conversation moved on to the subject of “designated sections”. During the conversation, the example of Swift was used and Josh felt the technical committee’s feedback indicated that either Swift in its entirety should be a designated section or that none of Swift would be considered binary. Josh also felt that the technical community (either the TC or PTLs) should be responsible for such decisions but I felt that while the TC can provide input, trademark policy decisions must ultimately be made by the board lest we taint the technical communities technical decision making by requiring significant political and business implications to be considered.

One element of clarity that emerged from the discussion was the simple point that “must have” tests were intended to drive interoperability while designated sections were intended to help build our community by requiring vendors to ship/deploy certain parts of the codebase and, by implication, contribute to those parts of the codebase.

As time ran short, the board voted to approve the selection criteria used by the DefCore committee. A straw-poll was also held to get a feel for whether board members saw the need for an “OpenStack compatible” mark in addition to the “OpenStack powered” mark. All but three of the board members (Monty, Todd and Josh) indicated their support for an additional “OpenStack compatible” mark.

Win The Enterprise

Briefly, Imad introduced the “Win the Enterprise” he an his team were kicking off with a session during the summit. The goal is to drive adoption of OpenStack in the enterprise by analyzing the technical and business gaps that may be hindering such adoption and coming up with an action plan to address them.

Feedback from board members was quite positive, with the discussion centered around how the group would measure their success and how they would ensure they operated in the most open and transparent way possible.

There was also some discussion about the need for more product management input to the project along with an additional focused effort on end-users of OpenStack clouds.

Wrapping Up

After the meeting drew to a close, board members joined members of the technical committee for a joint meeting. I’m hoping one of the awesome individuals on the technical committee will write a summary of that meeting!

This was a hugely draining week for many of us at Red Hat. As I prepare to completely switch off for a week, allow me to pass on this sage advise from Robyn Bergeron:

Keep Calm and Ride The Drama Llama

Heartbleed

Thursday, April 10th, 2014

Watching #heartbleed (aka CVE-2014-0160) fly by in my twitter stream this week, I keep wishing we could all just pause time for a couple of weeks and properly reflect on all the angles here.

Some of the things I’d love to have more time to dig into:

Mar 4 OpenStack Foundation Board Meeting

Wednesday, March 12th, 2014

On March 4th, the OpenStack Foundation Board of Directors met for an all-day, in-person meeting at DLA Piper’s office in Palo Alto, California. This my informal recollection of the meeting. It’s not an official record, etc.

Some 20 of the 24 board members managed to make the meeting in person with Todd and Tristan joining over the phone. Yujie Du and Chris Kemp were unable to attend.

As usual our teleconferencing capabilities were woefully inadequate for those hoping to contribute remotely. However, this time Rob and Lew joined a Google Hangout with video cameras trained on the meeting. One would hope that made it a little easier to engage with the meeting, but we didn’t really have any feedback on that.

Before we got started properly, Alan took some time to recommend “Startup Boards: Getting the Most Out of Your Board of Directors” to the directors (based, in turn, on Mark Radcliffe’s recommendation) as a book which could provide some useful background on the responsibilities of directors and what it takes to make a successful board.

[Update: Josh points out we also approved the minutes of the previous meeting]

Executive Director Update

Our first meaty topic was one of Jonathan’s regular updates.

Jonathan talked about the continued tremendous growth in interest around the project will all the foundation staff’s key metrics (e.g. website, developers, twitter, youtube, etc.) at lease doubling in 2013. One interesting aspect of this growth is that the China, India and Japan regions all grew their share of website traffic and this lead to a discussion around whether having the last summit in Hong Kong directly contributed to this shift.

Our community is growing too. We now have over 15,000 individual members of the foundation, over 2,000 contributors to the project over its lifetime and over 400 unique contributors to the project each month.

The mention of 15,000 individual members lead to a somewhat lengthy discussion about the fact that individual membership may be terminated under the following clause of the bylaws:

failure to vote in at least 50% of the votes for Individual Members within the prior twenty-four months unless the person does not respond within thirty (30) days of notice of such termination that the person wishes to continue to be an Individual Member

Jonathan explained how shortly after the foundation was launched, over 6,000 people signed up as individual members. Only 1,500 or so of those initial members have since voted in elections, so we could potentially be looking at removing somewhere in the region of 6,000 members in 2014. This reduced membership will facilitate bylaws changes by making it easier (or even possible) to reach the quorum necessary under clause 9.2(a):

requires an affirmative vote of a majority of the Individual Members voting as provided in Article III, but only if at least 25% of the Individual Members vote

Some discussion points around this included whether a future bylaws change should reduce the quorum requirement to something like 10%, that terminated members can re-register but would then have to wait 180 days in order to be eligible for voting and that project contributors need to be foundation members but some contributors may not be in the habit of voting and may have their membership terminated.

Jonathan moved on with his slide deck and briefly mentioned some of the foundation’s new supports like Oracle and Parallels. He also talked about OpenStack is increasingly fulfilling its role as a platform and is being put to work for many diverse use cases. He also included a slide with many positive media and analyst quotes about the project like “Industry support has coalesced around OpenStack”.

Jonathan then moved on to the foundation’s budget, describing it as an $8M budget which turned out to be $11M. Income was up, but expenses were kept in line such that $2.5M could be put in the bank. He expressed particular pride that 18 months ago the foundation was just starting with no money and already had built up a significant buffer which would allow us all to feel confident about the foundation’s future, even in more turbulent or unpredictable times.

Finally, Jonathan review the foundation staff’s priorities for 2014:

  1. Improve the software – whether that be continued investment by the foundation in the software development process or organizing activities which bring user feedback into the project
  2. Improve interoperability between OpenStack-powered products and services
  3. Grow the service provider global footprint, with a specific mention for interest from telco operators at Mobile World Congress around OpenStack and NFV

DefCore Update

Next up, Rob and Josh provided an update on the progress of the DefCore committee, requesting a checkpoint from the board as to whether there was consensus that the current approach should continue to be pursued.

Rob started by reviewing the purpose of DefCore and the approach taken to date. He explained that the committee is mandated to look at ways of governing commercial use of the OpenStack trademark and that some issues are deliberately being punted on for now, e.g. an API interoperability trademark and changes to the bylaws.

Josh took over and reviewed the currently agreed-upon criteria that will be used when evaluating whether a given capability will be required in order to use the trademark, e.g.

  1. Stable – required to be stable for >2 releases
  2. Complete – should be parity in capability tested across extension implementations
  3. Discoverable – e.g. can be found in Keystone and via service introspection
  4. Widely Deployed – favor capabilities that are supported by multiple public cloud providers and private cloud products
  5. Tools – supported by common tools
  6. Clients – part of common libraries
  7. Foundation – required by other must-have capabilities
  8. TC Future Direction – reflects future technical direction
  9. Documented – expected behaviour well documented
  10. Legacy – previously considered must-have
  11. Cluster – tests are available for this capability?
  12. Atomic – unique capability that cannot be built out of other must-have capabilities
  13. Non-Admin – capability does not require administrative rights

Next, Rob, Josh and Troy walked the board through a draft spreadsheet evaluating potential capabilities against those criteria.

Much of the subsequent discussion revolved around various board members being very eager to get wider feedback on the ramifications of this process, particularly around identifying the most thorny and controversial results. Concerns were expressed that the process has been so involved and detailed that few people are well appraised of where this is headed and may find some of the results very surprising.

Rob & Josh felt that this spreadsheet approach means that we can solicit much more targeted and useful feedback from stakeholders. For example, if a project feels one of its capabilities should be must-have, or a cloud provider is surprised that a capability it doesn’t yet provide is seen to be must-have, then the discussion can be around that specific capability, whether the set of criteria and weighting used for evaluation are appropriate, and whether the capability has been correctly evaluated against those criteria.

Finally, Rob & Josh explained the proposed approach for collecting the test results which would be used for evaluating trademark use applications. The idea (known as TCUP, or “tea-cup”, for “test collect, upload and publish”) currently being developed in the refstack repo on stackforge would allow people to download a docker container image, add your cloud credentials and endpoint URL, run the container which would then execute the tests against the endpoint and upload the results.

[Update: Josh points out that data uploaded via TCUP will "be treated as confidential for the time being"]

Driver Testing

Next, Boris Renski took the floor to talk about Nova, Neutron and Cinder driver testing particularly with a view to how it might relate to trademark usage. This relates to his blog post on the topic some weeks ago.

There were two main observations about changes happening in the technical community – (1) that projects were demanding that vendor maintainers provide reliable third-party automated testing feedback in gerrit patch reviews and (2) that manually maintained, often out-of-date, “driver compatibility matrices” in the OpenStack wiki may soon be replaced by dashboards showing the results of these third-party automated testing systems.

Boris wished to leave that technical work aside (since it is not the board’s domain) and focus the discussion on whether the board would consider a new trademark program such that vendors who’s drivers pass this automated testing would be allowed the use of a trademark such as “Built for OpenStack”.

The debate quickly got heated and went off in several different directions.

Part of the discussion revolved around the automated testing requirements projects were placing on projects, how that worked in practice, the ramifications of that, how deprecating drivers would work, whether a driver being in-tree implied a certain level of quality, etc. I felt the board was really off in the weeds on a topic that is under the authority of the individual projects and the TC. For example, it was easy to forget during the discussion that PTLs ultimately had the discretion to waive testing requirements for individual drivers.

Another surprising element to this was parallels being drawn with the overlap between the OpenStack Activity Board and Stackalytics. Some board members felt that Mirantis’s work on Stackalytics had deliberately duplicated and undermined the Activity Board effort, and that the same thing was happening here because this driver testing dashboard naturally (according to those board members) belongs under the DefCore/Refstack efforts. Boris acknowledged he “had his wrist slapped over Stackalytics” and was attempting to do the right thing here by getting advance buy-in. Others felt that the two efforts were either unrelated or that competing efforts can ultimately lead to a better end result.

Another thread of the discussion was that Boris’s use of, or allusion to, the special term “certification” automatically strayed this topic into the area of the OpenStack brand and that it was inappropriate to speculate that the foundation would embark on such a program before the board had discussed it.

In the end, the board directed that Jonathan should work with Boris and Rob on a plan to collect any automated test results out there and, secondly, work with the DefCore legal subcommittee to explore the possible use of the trademark in this context.

Operator’s Feedback

Tim Bell took the floor next to talk the board through feedback from OpenStack operators collected at the OpenStack Operators Mini Summit the previous day.

The etherpad linked above perhaps provides a better summary than I can give, but some of the highlights include:

  • The notion that operators should be able to provide feedback on blueprints as they are drafted, to help get operational insights to developers early on in the development process
  • Some observations on the stability (or lack thereof) of some of the core OpenStack components
  • The importance of a solid upgrades story was re-iterated
  • Some great feedback on TripleO
  • The split between teams doing CI/CD and those consuming releases
  • How to encourage operators to file more bugs upstream
  • Lots, lots more

This feedback was well received by the board and triggered a bunch of discussions and questions.

As a final point, Josh raised some concerns about how the invitee list was drawn up and how he felt it would have been appropriate for vendors (like Piston) to recommend some of their customers to be invited. Tim felt this was an unfair criticism and that the user committee had worked hard to seed the limited seating event with a diverse set of invitees before opening it up to the public.

Emerging Use Cases – NFV

Finally, with limited time remaining, Toby Ford from AT&T briefed the board on Network Function Virtualization (NFV) as an emerging use case for OpenStack which is heavily tied to SDN (Software Defined Network). He described how AT&T have set themselves a mission to:

Simplify, open up, and scale our Network to be more agile, elastic, secure, cost effective and fast by (a) moving from hardware centric to software centric, (b) separating the control plane and data plane and (c) making the network more programmable and open

Toby did a great job of walking through this complex area, leaving me with the understanding that there is a massive shift in the networking industry from hardware appliances to scale-out software appliances running on virtualized commodity hardware.

There appears to be consensus in the networking industry that OpenStack will be the management and orchestration platform for this new world order, but that there is a serious need for telcos and networking vendors to engage more closely with OpenStack in order to make this happen.

Wrap Up and Evening Event

Alan then wrapped up the meeting a little early after talking through the schedule for our next meetings with a conf call on April 3 and an in-person meeting on May 11.

The board then moved on to a local restaurant for dinner. Before and after dinner, I had some great conversations with Tim, Monty, Van and Troy. Funnily enough, because of the layout of the tables and the noise in the restaurant, it was only really possible to talk to the person sitting directly opposite you and so I found myself having an exclusive 2 hour dinner date with Boris! At one point, after Boris knocked a glass of wine over me, I joked that I should tweet “Red Hat and Mirantis tensions finally bubble over to physical violence”. But, in all honesty, these in-person, informal conversations around the board meetings are often far more effective at enabling shared understanding and real collaboration than the 20+ person meetings themselves. Such is the nature of the beast, I guess.

Naked Pings

Thursday, February 20th, 2014

Back in November 2009, ajax sent an email on IRC etiquette to Red Hat’s company-wide mailing list. I’ve had to refer several people to it over the years, so I asked ajax for permission to publish it. He agreed. Here it is in all its glory.

From: Adam Jackson
To: memo-list
Subject: On “ping” etiquitte
Date: Tue, 17 Nov 2009 12:21:30 -0500

IRC has developed a “ping” convention for getting someone’s attention. It works because most clients will highlight channels in which your name has been mentioned, so something like

ajax: ping

will make that channel show up pink instead of white for me [1].

I wish to correct, or at least amend, this behaviour. The naked ping should be Considered Harmful, for at least two reasons. The first is that it conveys no information. The recipient of your ping, like you, is a Busy Person. They may be in the middle of something requiring intricate thought, and should not be interrupted for anything less than fire, flood, or six figures of revenue. Worse, _you_ may forget why you pinged someone; when, four hours later, your victim gets back to IRC and responds to you, _you_ will be disrupted in turn trying to remember what was on your mind in the first place.

The second, more subtle reason proceeds from the first. A ping with no data is essentially a command. It’s passive-aggressive; it implies that the recipient’s time is less valuable than yours. [2] The pingee will respond in one (or both) of two ways. Either they will experience increased stress due to increased unpredictable demands on their time, or they will simply ignore naked pings.

The fundamental issue here is a misunderstanding of the medium. IRC is not a telephone. It’s volatile storage. The whole reason the ping works is because the client remembers seeing the ping, and can save it in your history buffer so you can see who was talking to you and why.

The naked ping removes this context.

Please. Save your time. Save my time. Make all of our lives more efficient and less stressful. Ping with data. At a minimum:

ajax: ping re bz 534027

See the difference? Now you’ve turned slow, lockstep, PIO-like interaction into smooth pipelineable DMA. It’s good for your hardware, and it’s good for you.

[1] – irssi 4 life.

[2] – Their time may well be less valuable than yours. That’s not the
point.

- ajax

Jan 30 OpenStack Foundation Board Meeting

Friday, February 7th, 2014

The OpenStack Foundation Board of Directors met for a two hour conference call last week. This was the first meeting of the board since the recent Individual and Gold member director elections.

As usual, this my informal recollection of the meeting. It’s not an official record, etc.

Your trusty reporter had just arrived in Brussels for FOSDEM and got to stay in his hotel room for this meeting rather than sampling Belgian’s fine beers. Oh, the sacrifices we make! :-P

Preliminaries

As usual, calling the meeting to order was a challenge and it was at least 15 minutes after the scheduled start before we completed the roll call.

Next, Alan welcomed our new directors:

  • Yujie Du
  • Alex Freedland
  • Vish Ishaya
  • Imad Sousou

and thanked our outgoing directors:

  • Nick Barcet
  • Hui Cheng
  • Joseph George
  • Lauren Sell

as well as thanking those directors who served on the board for part of 2013:

  • Devin Carlen
  • Jim Curry
  • John Igoe
  • Kyle MacDonald
  • Jon Mittelhauser

Policies, Communication Channels and Meetings Schedule

Since it’s a new year, we took the opportunity to review the various policies which apply to board members.

Josh went over our transparency policy mentioning that the board endeavours to be as transparent as possible, with board meetings open to the public, a summary of meetings posted to the foundation list and directors encouraged to use the foundation mailing list for discussions. Sub-committees of the board are expected to be similarly transparent, with wiki pages and public mailing lists. Some caveats to this policy are that board members are not allowed to make public comments about the board meeting until after Jonathan has posted his summary (or 72 hours have passed), members should not discuss executive sessions and the distribution of some non-public documents may have to be limited to directors.

Alan also mentioned our Code of Conduct and encouraged directors to read it carefully. Finally, Jeff from DLA Piper walked us through our antitrust policy where he emphasised the importance of avoiding even the perception that board members are coming together to advance the interests of some companies over others. Members should restrict themselves to pro-competitive collaboration.

Next, we quickly reviewed the various channels for communication that directors need to be aware of – webex for conf calls, the foundation and foundation-board mailing lists, the #openstack-board and #openstack-foundation IRC channels, informal etherpads that we use during board meetings and the various committee mailing lists.

Finally, we discussed the upcoming board meetings – an all-day face-to-face meeting in Palo Alto on March 4, a 2 hour conference call on April 3, an all-day face-to-face meeting in Atlanta on May 11 in advance of the Atlanta summit and another face-to-face on July 21 at OSCON.

The subject of the timing of the Atlanta face-to-face was raised again. May 11 is also Mother’s Day (in the US and some other countries) which is a nasty conflict for many board members. However, a poll amongst board members had already established that no better time around the summit could be found, so we are proceeding with the meeting on May 11. The question was raised about whether future board meetings should be scheduled to not align with our summits, but the objection to this idea was that it puts too much of a time and budget strain on those members who have to travel a long distance for the meetings.

Status Reports From Committees

Finally, time to move on to some more meaty topics! A member of each committee of the board was asked to provide a status update and plans for the year ahead.

Alan first described the work of the compensation committee who are responsible for defining and evaluating the goals and performance of the Executive Director. In summary, the committee concluded that Jonathan met his 2013 goals and new goals have been set for 2014.

Next up, Sean Roberts talked about the finance committee. This committee works with the foundation staff on financial budgeting and accounting. Sean described the foundation’s IRS filing and that the foundation’s 2012 financial audit has been completed and deemed clean (with a note that the foundation is “operating on a cash basis”). The foundation’s application for 501(c)(6) is progressing with the IRS asking for some clarifications which were returned to them in December. The committee meets monthly to review any discrepancies above 10%, but there have been no such issues so far. Essentially, everything is in excellent shape.

Tim Bell talked through the latest from the user committee. Tim mentioned the user survey that was published at the Hong Kong summit and how the committee has asked the TC for input on the kind of feedback that would be useful for developers. The committee is preparing to run another survey in advance of the Atlanta summit. Tim also mentioned that the user committee is running a couple of small, focused “operator mini-summits” over the next few months to bring operators together to share their feedback. Tim described the challenge running the committee with a small number of core volunteer members so as to ensure the privacy of survey results while also encouraging volunteers to help with tasks like turning survey feedback into blueprints for new features.

Van Lindberg gave an update on the legal affairs committee. He emphasised that the committee is not counsel for the foundation or the board, but rather a group which makes recommendations to the board on IP policy. He recapped on some of the patent policy recommendations from last year, for example that the foundation should join OIN. There was a brief mention of the fact that all the committee members are currently lawyers and the by-laws limits the number of members to five. He also mentioned that the DefCore committee has a related sub-committee examining possible by-laws changes.

Todd described the elections committee, that is was formed in February 2013 with 8 members and the goal to consider possible changes to the individual member election process. The committee is currently considering proposing a change to either Condorcet or STV and held a town-hall meeting in Hong Kong on the subject. Todd noted that the meeting was lightly attended and there generally has been rather low participation in the process. The main hurdle to getting such a change passed is that a majority of at least 25% of our individual members would need to vote for the by-laws change and the turnout for the previous election was only 17%. However, in July we will be able to begin making inactive members ineligible for voting and this should help us achieve the required turnout.

Rob gave an update from the DefCore committee[5] which is considering changes to the requirements for commercial implementations of OpenStack who wish to use the OpenStack trademark. The committee is currently working to identify a set of must-pass tests and the functional capabilities which these correspond to. Rob mentioned that some projects currently have no or minimal test coverage and, as a result, their capabilities could not be considered for inclusion in the requirements. Rob also mentioned a “programs vs projects” issue which had been identified during the committee discussions and that a meeting with the TC would be required to resolve the issue. I proposed that Rob and Josh could join the TC’s IRC meeting to discuss the issue.

Finally, Simon gave a brief overview of the work of the gold member application committee. This committee helps prospective gold members prepare their application such that it fully anticipates all the questions and concerns the board may have about the application.

Wrapping Up

While there were some small number of other items on the agenda, we had run out of time at this point. In the short time available, we had covered a broad range of topics but hadn’t really covered new ground. This meeting was mostly about rebooting the board for 2014.

The 2014 OpenStack Individual Member Director Elections and Red Hat

Wednesday, December 11th, 2013

tl;dr – the affiliation limit means that at most one of the two Red Hat affiliated candidates can be elected. The cumulative voting system makes it likely that both of us running seriously damages both of our chances of being elected. A preferential voting system like Condorcet or STV would not have this problem.

At Red Hat, those of us who contribute to OpenStack take very seriously our responsibility to put what’s good for the project first and foremost in our minds – to wear our “upstream hat”, as we like to say. That’s especially true for me and Russell Bryant.

However, now that the candidate list for the 2014 OpenStack Individual Member Director Elections has been finalized, we find ourselves wrestling with the fact that Russell and I are both running as candidates. Two aspects of our election system make this a problem. First, the cumulative voting system means that those who would be happy to vote for either me or Russell are forced to choose between us – essentially, we are damaging each others’ chances of being elected. Secondly, the affiliation limit means that even if we were both lucky enough to receive enough votes to be elected, one of us would be eliminated by the limit.

The combination of these two issues means that we have to factor our affiliation into our decision. The rules place affiliation front and centre in election system, even though Individual Member Directors are not elected to represent their employer.

Now, I’m personally guilty of not pushing this election system issue hard enough over this past year. At one point I favoured experimenting with a tweak to the cumulative system over a more dramatic change because I found the prospect of getting a majority of over 25% of our enormous electorate to vote in favour of a change so daunting. I want to be completely open about this decision we now face because I want to help raise awareness about how important an issue it is.

Given that Red Hat is a Platinum Member and has a automatic seat on the board, the options we’re weighing up are:

  1. Continue with both Russell and me on the ballot, accepting the risk that we’re damaging each others chances.
  2. I or Russell remove ourselves from the ballot, giving the other of us the best possible chance of being elected.
  3. Brian Stevens steps down from the board and I or Russell takes his place, giving whichever of us remains on the ballot the best possible chance of being elected.

It’s not an easy decision. We both feel we have something to offer on the board. Both of us would be very proud to be elected to represent the Individual Members. Both of us feel that Brian Stevens (our CTO who we greatly respect) is the best possible representative for Red Hat on the board.

We will make a decision on this before the election, but right now we don’t see any of the options as being particularly better than the others. But, at the very least, I hope everyone will find this useful as a concrete example of why our election system needs to change.

OpenStack “Core” and Interoperability

Wednesday, October 30th, 2013

I’ve been following the “what is core?” conversation since the “Incubation Update” committee completed its work some time ago. I’m really happy to see Rob, Alan and others put so much work into moving this along, but each time I try to catch up on progress I find myself a little bewildered.

Since it’s going to be a big topic during next week’s board meeting, I figured I’d try to get my thoughts together here.

What’s it all About?

Even the question bothers me – "what is core?". Why is that an important question?

One reason for its importance is that the bylaws say:

The Core OpenStack Project means the software modules [...] for which an OpenStack trademark may be used

In other words, Heat can't call itself "OpenStack Orchestration" unless the Board of Directors approve Heat as part of "The Core OpenStack Project". If that was all this was about, I'd be like "hell yes! Heat should be known as OpenStack Orchestration!". But, it's clearly not just about this, so I tend to ignore this aspect. I don’t actually think much of what is being discussed is all that relevant to this particular issue.

So, what else is this all about? Well, one clue is the emphasis on testing in Rob’s list of “10 core positions”.

OpenStack Core means passing all “must-pass” tests

This is an example of where I get really confused. Maybe I’m just getting hung up on language. I think we’re talking about cloud providers, distributions, vendors, deployers, etc. self-certifying themselves using a test suite in order that they can use the OpenStack trademark. But are we talking about labelling these self-certified clouds as “Core”? Aren’t we making this all very confusing by using the term “Core” both to describe the self-certified clouds/products and also to describe the subset of OpenStack they are required to include? I’d phrase this as:

The OpenStack Core definition includes a list of tests which certified clouds must pass

Anyway, I’m not going to nitpick my way through all of this. I think I understand the intent here, but I like to approach this from an entirely different angle.

A Market of Interoperable OpenStack Providers

(Yes, that is Simon Wardley’s terminology)

We all know that one of the basic goals of OpenStack is for there to be a bunch of public clouds available to users around the world and that interoperability between these public OpenStack clouds will make it easier for users to switch cloud providers and encourage competition between these providers.

To my mind, that’s what this whole “what is core?” conversation is really about. We want to use an OpenStack trademark program to help build this marketplace by enforcing interoperability between clouds which use the OpenStack trademark.

(And yes, I understand that the “what is core?” discussion is also related to questions like what is required of an OpenStack distribution. I’m focusing on the question of public cloud interoperability because I think it’s the most important. IMHO, the whole conversation is pointless unless it moves this particular topic along quickly.)

To do that, we need to define which APIs these clouds must expose, or as Rob puts it – which use cases these clouds must support. To take a simple example, should these clouds be required to expose the Glance API for uploading images?

And here’s where I get confused again. Why are we talking about “what is core?” when we could simply say “which APIs are required to be exposed by certified OpenStack clouds?”. Again, I’m trying not to be a pedant, but the way the question is framed leaves me unsure whether we’re really all talking about the same thing.

I think the focus should be on immediate baby-steps towards kick-starting this marketplace. One simple question – if and when we certify the first batch of interoperable clouds, would we rather have a smaller number of big clouds included or a large number of smaller clouds? In terms of resource capacity provided by this marketplace, I guess it’s more-or-less the same thing.

Let’s assume we absolutely want (at least) a small number of the bigger providers included in the program at launch. Let’s say “small number” equals “Rackspace and HP”. And let’s assume both of these providers are very keen to help get this program started. Isn’t the obvious next baby-step to get representatives of those two providers to figure out exactly what level of interoperability they already have and also what improvements to that they can make in the short term?

If we had that report, we could next do a quick “sniff test” comparing this to many of the other OpenStack clouds out there to make sure we haven’t just picked two clouds with an unusual set of compatible APIs. Then the board could make a call on whether this represents a reasonable starting point for the requirements of OpenStack clouds.

No, this isn’t perfect. But it would be a genuine step forward towards being able to certify some clouds. We would have some data on commonalities and have made a policy decision on what commonalities are required. It would be a starting point.

OpenStack Compatible?

The next big decision for the board is whether a cloud which uses the OpenStack trademark must actually be a deployment of OpenStack’s code. Is this a question of “OpenStack clouds” or “OpenStack compatible clouds”?

I do think it would be damaging to OpenStack if this marketplace took off and was dominated by providers which don’t use or contribute to OpenStack. But, as Rob says, the trademark isn’t the only way of avoiding this situation – there’s also our “velocity and culture”.

I struggle to see how trademark requirements around the use of OpenStack code would work. How do you define which code must be used? Must that code be used unmodified? If not, how much can you change? What does it even mean to “use” a piece of code? That user requests must be executed using that code? Maybe this would just be a “yes, we use and contribute to OpenStack” good faith statement which the Foundation would make a judgment call on? If so, how do we ensure transparency and fairness around how that call is made?

If this question proves to be a stumbling block, I’d prefer to see an “OpenStack compatible cloud” trademark program established quickly. Getting these interoperability guarantees in place for our users takes priority over ensuring that certified providers actually use and contribute to OpenStack.

Conclusion of Sorts

I think our immediate concern should be kicking off a trademark program for certifying interoperability between OpenStack clouds. I’m frustrated whenever I think this “what is core?” discussion is tackling questions which aren’t immediate blockers to making progress on this program.

The board has to make (or oversee the making of) some policy questions.

Firstly, what APIs are required in OpenStack™ clouds? I’d favour starting to answer this by first looking at the current interoperability levels between existing clouds.

Secondly, whether and how we insist on the use of OpenStack code in OpenStack™ clouds. If this turns out to be a difficult problem, then I think we should just start with an “OpenStack™ compatible cloud” program.

A New OpenStack Technical Committee

Friday, October 18th, 2013

The results of the OpenStack Technical Committee have just been announced. I’m very grateful to everyone who voted for me. Thanks!

This is going to be a very different committee than previously. For the first time, PTLs of integrated projects do not have an automatic seat. I think this will be a positive change and see TC members generally take more of an interest in cross-project issues. That said, we need to be careful to explicitly include PTLs in decision making which affect their projects.

One thing I’m curious about is how this new TC makeup will affect discussions about the increase in OpenStack’s technical scope. I went through each of candidate’s nomination emails and pulled out some quotes below. To me, this represents a high level of consensus towards a continued cautious and measured growth in the project’s scope … but make your own mind up! I’m actually quite surprised how many candidates felt it was important to give their views on this.

Monty Taylor

I have an expansive view of the scope of OpenStack. I do not think that ‘pure IaaS’ as a limiting factor in the definition serves or will serve any of our users. I think that instead of trying to come up with random or theoretical labels and then keeping ourselves inside of the pre-defined label, we should focus on writing software that solves problems for users.

Russell Bryant

One of the most exciting things for me over the last year has been helping to expand OpenStack by including a number of new projects. I think this growth is important for OpenStack’s success. I would like to continue to help guide this growth and to help figure out how OpenStack can scale as an organization and still be effective.

Anne Gentle

My goal is to provide an inclusive and supportive environment for projects while making OpenStack better for users and admins all the time. We are so fortunate to have the explosive growth and interest in OpenStack, and I want it to continue. We have built upon incredible ideas and I want us to be empowered to innovate.

Mark McLoughlin

We welcomed Heat, Ceilometer, Trove, Savannah, Marconi into the OpenStack family either as integrated or incubating projects. The TC carefully considered each of these applications and my own rule of thumb was “does it have a healthy contributor community and is it a sensible growth of OpenStack’s scope?”. I love to see this sustainable growth in our project and community.

Doug Hellmann

I share the view of many of the other candidates that OpenStack should not limit itself to today’s definition of IaaS. The history of computing is a progression of different levels of abstraction, and what we consider “platform” today may become “infrastructure” tomorrow.

Sean Dague

I’m incredibly excited by OpenStack’s growth (in people, code, scope), which I attribute to an incredibly welcoming and constructive community, and the velocity we get out of our preemptive integration system. As a TC member I’d do my best to ensure those conditions remain. I think we’ve only just begun to see what OpenStack will become [..]

James E. Blair

As a significant OpenStack user, I’m excited about the direction that OpenStack is heading. I’m glad that we’re accepting new programs that expand the scope of our project to make it more useful for everyone. I believe a major function of the Technical Committee is to curate and shepherd new programs through the incubation process. However, I believe that it should be more involved than it has been. We have been very quick to promote out of integration some exciting new projects that may not have been fully integrated. As a member of the TC, I support our continued growth, and I want to make sure that the ties that hold our collection of projects together are strong, and more than just a marketing umbrella.

Michael Still

First off, the TC has incubated a number of projects in the Havana release, and I’d like to see that continue. I think its important that we build a platform that includes the services that a deployer would
need to build a cloud and that those platform elements work well together. Now, its clear that not everyone will deploy all of the projects we are incubating, but I think its still important that they play well together and have a consistent look and feel.

John Griffith

New projects and growth are important to OpenStack however I don’t think that uncontrolled and disjointed growth in the form of new projects is a good thing, in fact I think it’s detrimental to OpenStack as a whole. I personally would like to see the TC have more involvement in terms of recommending/investigating new projects before they’re proposed or started by others. By the same token, I’d also like to see the TC take a more active role in the projects we currently have and how they all tie
together. I personally believe that having 10 or so individual projects operating in their own silos is not the right direction. My opinion here does NOT equate to “more control”, but instead should equate to being more helpful. With the continued growth of OpenStack I believe it’s critical to have some sort of vision and some resources that have a deep understanding of the entire eco-system.

Mark McClain

The issue of scope was a recurring theme during my recent term on the TC. As the OpenStack ecosystem grows beyond Infrastructure as a Service, the committee needs to more clearly define the criteria used to determine the kind of projects and programs that fit within the scope of integrated releases and how they move through the progression of incubation to graduation. In addition to defining the criteria, the Technical Committee should to work develop policies and procedures to provide some guidance to projects which are outside of the scope an integrated release, but valuable to our community.

Robert Collins

(I don’t see a directly relevant quote from Robert)

Oct 3rd OpenStack Foundation Board Meeting

Thursday, October 3rd, 2013

On Oct 3rd, the OpenStack Foundation Board held a two hour conference call meeting which was open to anyone to join. The agenda was published in advance.

As usual, this my informal recollection of the meeting. It’s not an official record, etc. Jonathan published an official summary on the foundation mailing list.

Preliminaries

Our chairman, Alan Clark began the meeting by holding a roll call. If my count was correct, we had 21 of the 24 members of the board on the call, which is a pretty good turnout.

Alan introduced Chris Kemp as a replacement for Devin Carlen who had been representing Nebula and the Gold Members. Chris briefly explained that changes in his role at Nebula has left him more time for “strategic stuff” and is delighted to spend more time with the OpenStack community and join the board. He thanked everyone for their work in progressing the foundation to date.

Before getting down to business, we quickly voted to approve the minutes of the previous meeting.

Executive Director Report

Jonathan Bryce took the floor and gave a really excellent update on the Foundation’s progress.

First off, he described the launch of the OpenStack Foundation Training Marketplace at LinuxCon in New Orleans. The site now has 12 companies listing training events at over 30 cities around the world and has seen over 10,000 unique visitors over the past 2 weeks. He reiterated how one of the Foundation’s stated priorities for this year is to help training providers get the word out on their education programs and thereby expand the pool of people with OpenStack skills.

Next up, Jonathan gave an update on preparations for the OpenStack Summit in Hong Kong. Everything is coming together nicely and the finalized speaking agenda has now been posted. He explained that, while they had received some 250 submissions for the summit in Portland, there were over 600 submissions for Hong Kong and yet not many extra speaking slots. The track chairs had a tough job whittling down the submissions but he thinks the end result is an agenda packed with great sessions. Jonathan warned that, unlike previous summits, we have a hard limit on the number of attendees and the “full access” passes will sell out in the next week or so.

Eileen asked exactly how many full access passes were issued and Jonathan explained there are 2,000 but that there are many more passes for the general sessions and expo hall. Joseph asked for some insight into the demographics of the registered attendees and how it compares to previous summits. Jonathan felt that we would see a large number of attendees from the region and that, while some companies may have reduced their numbers travelling from further afield, there would still be plenty of representation from all of the companies we are used to seeing at summits.

The discussion then moved on to looking ahead to 2014 and a sneak peak of next year’s budget. Jonathan explained how 2013 was a great year financially – particularly because the summits were larger than expected – and the Foundation goes into 2014 with a healthy budget surplus of over $1M. We expect to continue to see large summits and this will lead to increased revenues and expenses. Jonathan expects the Foundation to continue to invest in technology and infrastructure over 2014. He described the Foundation funded projects in 2013 to improve the member database and tie it to the contributor agree process and an improved system for handling summit speaker submissions.

Rob Hirschfeld asked whether the Foundation had budgeted for certification testing infrastructure being proposed as part of the “what is core” discussion. Josh felt this it not be necessary for the Foundation to fund this, and there was a brief data tracking and authentication systems.

Jonathan continued by describing the rapid growth in Foundation staff during 2013, but that he expects ongoing staffing costs to have mostly levelled off for 2014. He went on to talk about how he sees the role of the Foundation over the next year to be around connecting different parts of the community, helping to gather data on users and deployments and sharing information with the world about new features in each release.

One Jonathan had finished, Lew Tucker congratulated Jonathan on the progress and asked whether a written summary could be prepared for board members so that we could help get the word out. Josh pointed out that we were soon due an annual report which would contain much of this information.

Josh then asked a question about the project budget for gold member fees and how many new members this assumed would be added in 2014. There was some confusion on the issue because gold member fees are based on the member’s revenue, but the conclusion was that the addition of 4 new members was projected. This would bring the total number of gold members up to 20 out of a total possible 24 members.

Election Committee Report

Next up, Todd Moore delivered an excellent summary of the discussions and conclusions of the committee established to consider whether any changes were need for the individual and gold member director election systems and what those changes might be.

Todd’s report – which will be published in full soon – described how we considered three separate questions. Firstly, whether and how concerns about the fairness of the individual member director election system (particularly the issue of large blocks of affiliated voters) should be addressed, secondly whether we should consider any rule changes around foundation employees eligibility for the individual member director election and, finally, whether the eligibility rules for the 2013 gold member director elections had been properly enforced.

The last topic was easily dispensed with – Todd and the committee worked with the foundation staff to verify that all eligibility rules (i.e. eligibility as a candidate and eligibility to vote) had been properly enforced.

The first topic proved to be substantially more contentious. The pros and cons of changing to an STV based election system or tweaking the rules of the current system to halve (from 8 to 4) the number of votes which you could allocate to a single candidate were explained in great detail. Todd explained how the committee felt that changes were not necessarily warranted but that it would be wise to seriously consider the option of the “max 4 votes per candidate” tweak.

A motion was proposed to not fundamentally change the voting system and was almost voted for until it became apparent that Monty wanted to speak but was unable to come off mute. Once that was sorted out, Monty stated that the cumulative system was utterly broken and that block voting did, in fact, influence the outcome of the election. He felt strongly that a change to STV was needed. The debate then took off in earnest with many well put points made by various members. I explained how I agreed fully with Monty but that I was convinced by the “it’s working reasonably well; this doesn’t warrant a the disruption of a bylaws change” but that we needed to be alert to any renewed concern from members after the next election. In the end the vote was passed with no abstentions and Monty voting against.

The discussion then moved on to the question of the “only 4 votes per candidate” tweak. Much of the discussion hinged on whether this tweak required a bylaws change. The election committee and Jonathan felt that it was compatible with the “cumulative voting system” language in in section 3.9a of the bylaws but legal counsel for the foundation (Mark Radcliffe) disagreed strongly and felt “cumulative voting” was “binary in the eyes of the law”. Since the committee had suggested this tweak on the understanding that it was easy to implement, the idea was dropped.

Next, we discussed the second issue – the question of foundation employees on the board. The committee did not have a consensus recommendation on this but Todd did summarize the different viewpoints and arguments. One approach proposed that the executive director would be granted a new, automatic seat on the board and (because there is a limit of two members per organization), only one additional foundation employee could be elected to the board rather than the current limit of two employees. There was lots of debate and I felt Jonathan’s input was particularly interesting – that he did not see the need for an automatic executive director seat and he had reservations about restricting employees from being directors since they could bring invaluable and passionate input, should the electorate choose to elect them. In the end, it was felt there was not sufficient cause to warrant a disruptive bylaws change and the motion was withdrawn.

So, in summary, there will be no election system changes proposed to the electorate during this cycle. However, the board again re-stated the importance that all members adhere to the community code of conduct which states:

Respect the election process. Members should not attempt to manipulate election results. Open debate is welcome, but vote trading, ballot stuffing and other forms of abuse are not acceptable.

Finally, the committee took an action item from the board to analyze the results of the next election and provide the board with a report on the potential effect of any of the options proposed.

Wrapping Up

Because of the lengthy debate on the election system, we had run out of time and the rest of the agenda had to be postponed until the in person board meeting in Hong Kong.

Edits

  • Changed from “legal counsel for the board” to “legal counsel for the foundation”. Thanks to Richard Fontana for the correction.
  • Included a reference to the code of comment. Thanks to Rob Esker for his question.