OpenStack “Core” and Interoperability

I’ve been following the “what is core?” conversation since the “Incubation Update” committee completed its work some time ago. I’m really happy to see Rob, Alan and others put so much work into moving this along, but each time I try to catch up on progress I find myself a little bewildered.

Since it’s going to be a big topic during next week’s board meeting, I figured I’d try to get my thoughts together here.

What’s it all About?

Even the question bothers me – "what is core?". Why is that an important question?

One reason for its importance is that the bylaws say:

The Core OpenStack Project means the software modules […] for which an OpenStack trademark may be used

In other words, Heat can't call itself "OpenStack Orchestration" unless the Board of Directors approve Heat as part of "The Core OpenStack Project". If that was all this was about, I'd be like "hell yes! Heat should be known as OpenStack Orchestration!". But, it's clearly not just about this, so I tend to ignore this aspect. I don’t actually think much of what is being discussed is all that relevant to this particular issue.

So, what else is this all about? Well, one clue is the emphasis on testing in Rob’s list of “10 core positions”.

OpenStack Core means passing all “must-pass” tests

This is an example of where I get really confused. Maybe I’m just getting hung up on language. I think we’re talking about cloud providers, distributions, vendors, deployers, etc. self-certifying themselves using a test suite in order that they can use the OpenStack trademark. But are we talking about labelling these self-certified clouds as “Core”? Aren’t we making this all very confusing by using the term “Core” both to describe the self-certified clouds/products and also to describe the subset of OpenStack they are required to include? I’d phrase this as:

The OpenStack Core definition includes a list of tests which certified clouds must pass

Anyway, I’m not going to nitpick my way through all of this. I think I understand the intent here, but I like to approach this from an entirely different angle.

A Market of Interoperable OpenStack Providers

(Yes, that is Simon Wardley’s terminology)

We all know that one of the basic goals of OpenStack is for there to be a bunch of public clouds available to users around the world and that interoperability between these public OpenStack clouds will make it easier for users to switch cloud providers and encourage competition between these providers.

To my mind, that’s what this whole “what is core?” conversation is really about. We want to use an OpenStack trademark program to help build this marketplace by enforcing interoperability between clouds which use the OpenStack trademark.

(And yes, I understand that the “what is core?” discussion is also related to questions like what is required of an OpenStack distribution. I’m focusing on the question of public cloud interoperability because I think it’s the most important. IMHO, the whole conversation is pointless unless it moves this particular topic along quickly.)

To do that, we need to define which APIs these clouds must expose, or as Rob puts it – which use cases these clouds must support. To take a simple example, should these clouds be required to expose the Glance API for uploading images?

And here’s where I get confused again. Why are we talking about “what is core?” when we could simply say “which APIs are required to be exposed by certified OpenStack clouds?”. Again, I’m trying not to be a pedant, but the way the question is framed leaves me unsure whether we’re really all talking about the same thing.

I think the focus should be on immediate baby-steps towards kick-starting this marketplace. One simple question – if and when we certify the first batch of interoperable clouds, would we rather have a smaller number of big clouds included or a large number of smaller clouds? In terms of resource capacity provided by this marketplace, I guess it’s more-or-less the same thing.

Let’s assume we absolutely want (at least) a small number of the bigger providers included in the program at launch. Let’s say “small number” equals “Rackspace and HP”. And let’s assume both of these providers are very keen to help get this program started. Isn’t the obvious next baby-step to get representatives of those two providers to figure out exactly what level of interoperability they already have and also what improvements to that they can make in the short term?

If we had that report, we could next do a quick “sniff test” comparing this to many of the other OpenStack clouds out there to make sure we haven’t just picked two clouds with an unusual set of compatible APIs. Then the board could make a call on whether this represents a reasonable starting point for the requirements of OpenStack clouds.

No, this isn’t perfect. But it would be a genuine step forward towards being able to certify some clouds. We would have some data on commonalities and have made a policy decision on what commonalities are required. It would be a starting point.

OpenStack Compatible?

The next big decision for the board is whether a cloud which uses the OpenStack trademark must actually be a deployment of OpenStack’s code. Is this a question of “OpenStack clouds” or “OpenStack compatible clouds”?

I do think it would be damaging to OpenStack if this marketplace took off and was dominated by providers which don’t use or contribute to OpenStack. But, as Rob says, the trademark isn’t the only way of avoiding this situation – there’s also our “velocity and culture”.

I struggle to see how trademark requirements around the use of OpenStack code would work. How do you define which code must be used? Must that code be used unmodified? If not, how much can you change? What does it even mean to “use” a piece of code? That user requests must be executed using that code? Maybe this would just be a “yes, we use and contribute to OpenStack” good faith statement which the Foundation would make a judgment call on? If so, how do we ensure transparency and fairness around how that call is made?

If this question proves to be a stumbling block, I’d prefer to see an “OpenStack compatible cloud” trademark program established quickly. Getting these interoperability guarantees in place for our users takes priority over ensuring that certified providers actually use and contribute to OpenStack.

Conclusion of Sorts

I think our immediate concern should be kicking off a trademark program for certifying interoperability between OpenStack clouds. I’m frustrated whenever I think this “what is core?” discussion is tackling questions which aren’t immediate blockers to making progress on this program.

The board has to make (or oversee the making of) some policy questions.

Firstly, what APIs are required in OpenStack™ clouds? I’d favour starting to answer this by first looking at the current interoperability levels between existing clouds.

Secondly, whether and how we insist on the use of OpenStack code in OpenStack™ clouds. If this turns out to be a difficult problem, then I think we should just start with an “OpenStack™ compatible cloud” program.

A New OpenStack Technical Committee

The results of the OpenStack Technical Committee have just been announced. I’m very grateful to everyone who voted for me. Thanks!

This is going to be a very different committee than previously. For the first time, PTLs of integrated projects do not have an automatic seat. I think this will be a positive change and see TC members generally take more of an interest in cross-project issues. That said, we need to be careful to explicitly include PTLs in decision making which affect their projects.

One thing I’m curious about is how this new TC makeup will affect discussions about the increase in OpenStack’s technical scope. I went through each of candidate’s nomination emails and pulled out some quotes below. To me, this represents a high level of consensus towards a continued cautious and measured growth in the project’s scope … but make your own mind up! I’m actually quite surprised how many candidates felt it was important to give their views on this.

Monty Taylor

I have an expansive view of the scope of OpenStack. I do not think that ‘pure IaaS’ as a limiting factor in the definition serves or will serve any of our users. I think that instead of trying to come up with random or theoretical labels and then keeping ourselves inside of the pre-defined label, we should focus on writing software that solves problems for users.

Russell Bryant

One of the most exciting things for me over the last year has been helping to expand OpenStack by including a number of new projects. I think this growth is important for OpenStack’s success. I would like to continue to help guide this growth and to help figure out how OpenStack can scale as an organization and still be effective.

Anne Gentle

My goal is to provide an inclusive and supportive environment for projects while making OpenStack better for users and admins all the time. We are so fortunate to have the explosive growth and interest in OpenStack, and I want it to continue. We have built upon incredible ideas and I want us to be empowered to innovate.

Mark McLoughlin

We welcomed Heat, Ceilometer, Trove, Savannah, Marconi into the OpenStack family either as integrated or incubating projects. The TC carefully considered each of these applications and my own rule of thumb was “does it have a healthy contributor community and is it a sensible growth of OpenStack’s scope?”. I love to see this sustainable growth in our project and community.

Doug Hellmann

I share the view of many of the other candidates that OpenStack should not limit itself to today’s definition of IaaS. The history of computing is a progression of different levels of abstraction, and what we consider “platform” today may become “infrastructure” tomorrow.

Sean Dague

I’m incredibly excited by OpenStack’s growth (in people, code, scope), which I attribute to an incredibly welcoming and constructive community, and the velocity we get out of our preemptive integration system. As a TC member I’d do my best to ensure those conditions remain. I think we’ve only just begun to see what OpenStack will become [..]

James E. Blair

As a significant OpenStack user, I’m excited about the direction that OpenStack is heading. I’m glad that we’re accepting new programs that expand the scope of our project to make it more useful for everyone. I believe a major function of the Technical Committee is to curate and shepherd new programs through the incubation process. However, I believe that it should be more involved than it has been. We have been very quick to promote out of integration some exciting new projects that may not have been fully integrated. As a member of the TC, I support our continued growth, and I want to make sure that the ties that hold our collection of projects together are strong, and more than just a marketing umbrella.

Michael Still

First off, the TC has incubated a number of projects in the Havana release, and I’d like to see that continue. I think its important that we build a platform that includes the services that a deployer would
need to build a cloud and that those platform elements work well together. Now, its clear that not everyone will deploy all of the projects we are incubating, but I think its still important that they play well together and have a consistent look and feel.

John Griffith

New projects and growth are important to OpenStack however I don’t think that uncontrolled and disjointed growth in the form of new projects is a good thing, in fact I think it’s detrimental to OpenStack as a whole. I personally would like to see the TC have more involvement in terms of recommending/investigating new projects before they’re proposed or started by others. By the same token, I’d also like to see the TC take a more active role in the projects we currently have and how they all tie
together. I personally believe that having 10 or so individual projects operating in their own silos is not the right direction. My opinion here does NOT equate to “more control”, but instead should equate to being more helpful. With the continued growth of OpenStack I believe it’s critical to have some sort of vision and some resources that have a deep understanding of the entire eco-system.

Mark McClain

The issue of scope was a recurring theme during my recent term on the TC. As the OpenStack ecosystem grows beyond Infrastructure as a Service, the committee needs to more clearly define the criteria used to determine the kind of projects and programs that fit within the scope of integrated releases and how they move through the progression of incubation to graduation. In addition to defining the criteria, the Technical Committee should to work develop policies and procedures to provide some guidance to projects which are outside of the scope an integrated release, but valuable to our community.

Robert Collins

(I don’t see a directly relevant quote from Robert)

Oct 3rd OpenStack Foundation Board Meeting

On Oct 3rd, the OpenStack Foundation Board held a two hour conference call meeting which was open to anyone to join. The agenda was published in advance.

As usual, this my informal recollection of the meeting. It’s not an official record, etc. Jonathan published an official summary on the foundation mailing list.

Preliminaries

Our chairman, Alan Clark began the meeting by holding a roll call. If my count was correct, we had 21 of the 24 members of the board on the call, which is a pretty good turnout.

Alan introduced Chris Kemp as a replacement for Devin Carlen who had been representing Nebula and the Gold Members. Chris briefly explained that changes in his role at Nebula has left him more time for “strategic stuff” and is delighted to spend more time with the OpenStack community and join the board. He thanked everyone for their work in progressing the foundation to date.

Before getting down to business, we quickly voted to approve the minutes of the previous meeting.

Executive Director Report

Jonathan Bryce took the floor and gave a really excellent update on the Foundation’s progress.

First off, he described the launch of the OpenStack Foundation Training Marketplace at LinuxCon in New Orleans. The site now has 12 companies listing training events at over 30 cities around the world and has seen over 10,000 unique visitors over the past 2 weeks. He reiterated how one of the Foundation’s stated priorities for this year is to help training providers get the word out on their education programs and thereby expand the pool of people with OpenStack skills.

Next up, Jonathan gave an update on preparations for the OpenStack Summit in Hong Kong. Everything is coming together nicely and the finalized speaking agenda has now been posted. He explained that, while they had received some 250 submissions for the summit in Portland, there were over 600 submissions for Hong Kong and yet not many extra speaking slots. The track chairs had a tough job whittling down the submissions but he thinks the end result is an agenda packed with great sessions. Jonathan warned that, unlike previous summits, we have a hard limit on the number of attendees and the “full access” passes will sell out in the next week or so.

Eileen asked exactly how many full access passes were issued and Jonathan explained there are 2,000 but that there are many more passes for the general sessions and expo hall. Joseph asked for some insight into the demographics of the registered attendees and how it compares to previous summits. Jonathan felt that we would see a large number of attendees from the region and that, while some companies may have reduced their numbers travelling from further afield, there would still be plenty of representation from all of the companies we are used to seeing at summits.

The discussion then moved on to looking ahead to 2014 and a sneak peak of next year’s budget. Jonathan explained how 2013 was a great year financially – particularly because the summits were larger than expected – and the Foundation goes into 2014 with a healthy budget surplus of over $1M. We expect to continue to see large summits and this will lead to increased revenues and expenses. Jonathan expects the Foundation to continue to invest in technology and infrastructure over 2014. He described the Foundation funded projects in 2013 to improve the member database and tie it to the contributor agree process and an improved system for handling summit speaker submissions.

Rob Hirschfeld asked whether the Foundation had budgeted for certification testing infrastructure being proposed as part of the “what is core” discussion. Josh felt this it not be necessary for the Foundation to fund this, and there was a brief data tracking and authentication systems.

Jonathan continued by describing the rapid growth in Foundation staff during 2013, but that he expects ongoing staffing costs to have mostly levelled off for 2014. He went on to talk about how he sees the role of the Foundation over the next year to be around connecting different parts of the community, helping to gather data on users and deployments and sharing information with the world about new features in each release.

One Jonathan had finished, Lew Tucker congratulated Jonathan on the progress and asked whether a written summary could be prepared for board members so that we could help get the word out. Josh pointed out that we were soon due an annual report which would contain much of this information.

Josh then asked a question about the project budget for gold member fees and how many new members this assumed would be added in 2014. There was some confusion on the issue because gold member fees are based on the member’s revenue, but the conclusion was that the addition of 4 new members was projected. This would bring the total number of gold members up to 20 out of a total possible 24 members.

Election Committee Report

Next up, Todd Moore delivered an excellent summary of the discussions and conclusions of the committee established to consider whether any changes were need for the individual and gold member director election systems and what those changes might be.

Todd’s report – which will be published in full soon – described how we considered three separate questions. Firstly, whether and how concerns about the fairness of the individual member director election system (particularly the issue of large blocks of affiliated voters) should be addressed, secondly whether we should consider any rule changes around foundation employees eligibility for the individual member director election and, finally, whether the eligibility rules for the 2013 gold member director elections had been properly enforced.

The last topic was easily dispensed with – Todd and the committee worked with the foundation staff to verify that all eligibility rules (i.e. eligibility as a candidate and eligibility to vote) had been properly enforced.

The first topic proved to be substantially more contentious. The pros and cons of changing to an STV based election system or tweaking the rules of the current system to halve (from 8 to 4) the number of votes which you could allocate to a single candidate were explained in great detail. Todd explained how the committee felt that changes were not necessarily warranted but that it would be wise to seriously consider the option of the “max 4 votes per candidate” tweak.

A motion was proposed to not fundamentally change the voting system and was almost voted for until it became apparent that Monty wanted to speak but was unable to come off mute. Once that was sorted out, Monty stated that the cumulative system was utterly broken and that block voting did, in fact, influence the outcome of the election. He felt strongly that a change to STV was needed. The debate then took off in earnest with many well put points made by various members. I explained how I agreed fully with Monty but that I was convinced by the “it’s working reasonably well; this doesn’t warrant a the disruption of a bylaws change” but that we needed to be alert to any renewed concern from members after the next election. In the end the vote was passed with no abstentions and Monty voting against.

The discussion then moved on to the question of the “only 4 votes per candidate” tweak. Much of the discussion hinged on whether this tweak required a bylaws change. The election committee and Jonathan felt that it was compatible with the “cumulative voting system” language in in section 3.9a of the bylaws but legal counsel for the foundation (Mark Radcliffe) disagreed strongly and felt “cumulative voting” was “binary in the eyes of the law”. Since the committee had suggested this tweak on the understanding that it was easy to implement, the idea was dropped.

Next, we discussed the second issue – the question of foundation employees on the board. The committee did not have a consensus recommendation on this but Todd did summarize the different viewpoints and arguments. One approach proposed that the executive director would be granted a new, automatic seat on the board and (because there is a limit of two members per organization), only one additional foundation employee could be elected to the board rather than the current limit of two employees. There was lots of debate and I felt Jonathan’s input was particularly interesting – that he did not see the need for an automatic executive director seat and he had reservations about restricting employees from being directors since they could bring invaluable and passionate input, should the electorate choose to elect them. In the end, it was felt there was not sufficient cause to warrant a disruptive bylaws change and the motion was withdrawn.

So, in summary, there will be no election system changes proposed to the electorate during this cycle. However, the board again re-stated the importance that all members adhere to the community code of conduct which states:

Respect the election process. Members should not attempt to manipulate election results. Open debate is welcome, but vote trading, ballot stuffing and other forms of abuse are not acceptable.

Finally, the committee took an action item from the board to analyze the results of the next election and provide the board with a report on the potential effect of any of the options proposed.

Wrapping Up

Because of the lengthy debate on the election system, we had run out of time and the rest of the agenda had to be postponed until the in person board meeting in Hong Kong.

Edits

  • Changed from “legal counsel for the board” to “legal counsel for the foundation”. Thanks to Richard Fontana for the correction.
  • Included a reference to the code of comment. Thanks to Rob Esker for his question.

Aug 6th OpenStack Foundation Board Meeting

On Aug 6th, the OpenStack Foundation Board held a two hour conference call meeting which was open to anyone to join. The agenda was published in advance.

As usual, this my informal recollection of the meeting. It’s not an official record, etc. Jonathan published an official summary on the foundation mailing list.

Preliminaries

Alan Clark chaired the meeting and spent the first few minutes reviewing the agenda and holding the roll call. We had more than 50% of the board members, and so had sufficient quorum.

We then reviewed the minutes of the previous meeting and approved them by vote. Todd, Troy, Nick, Devin and Jim all abstained because they weren’t present at that meeting.

Patent Cooperation

Next up was a follow-on from the patent cooperation discussion at our previous meeting.

The Legal Affairs Committee had previously proposed three options that we make no change, that we implement a Google-style patent pledge or that we introduce an OIN-style cross licensing agreement.

Keith Bergelt, CEO of the Open Invention Network, attended the meeting and gave a presentation describing the OIN and how it could play a key role in OpenStack’s strategy to create an environment of patent non-aggression around OpenStack. Keith has been actively working with the Legal Affairs Committee to develop a fourth proposal for the board which would make the best possible use of the OIN.

Keith described how the OIN was formed 8 years ago by IBM, NEC, Novell, Philips, Red Hat and Sony to ensure “freedom of action, freedom to operate” around Linux. It has some 560 member companies who sign up to a broad cross-licensing agreement whereby any patents they hold which read on Linux is licensed to all other member companies.

The patents which are covered by the OIN agreement are those which read on the software packages listed in the OIN’s Linux System Definition. Over time, this definition has grown to include the likes of Android, WebOS and, more recently, OpenStack. The OIN views OpenStack has one of the most import projects after Linux and feels that including OpenStack in its defintiion is an obvious step.

That OpenStack is included in the OIN system definition means that there are already 560 licensees committed to non-aggression around OpenStack. Keith feels that this goes a long way to achieving the OpenStack Foundation’s goals in this area and that we should seek to make the most effective use of the existence of this agreement rather than reinvent the wheel.

Keith proposes a more formalized relationship with the Foundation, that members would be encouraged to join the OIN, that we’d do joint press releases and that there would be an ongoing direct relationship with the TC to ensure the system definition is regularly updated to include all OpenStack projects.

Legal Affairs Committee

Van Lindberg was up next to provde an update from the Legal Affairs Committee but this turned out to be a continuation of the previous discussion on patent cooperation.

Van summed up this fourth proposal succinctly – that OpenStack members get behind the OIN. He went on to describe how a difficulty with this approach is that some members of OpenStack will never be members of the OIN because the OIN fundamentally see those companies as a threat to Linux. He didn’t name these companies.

So, the question becomes how we can ensure the ability of those OpenStack members who aren’t members of the OIN to join a patent cooperation initiative specifically around OpenStack.

Van first described how OpenStack LLC (the Rackspace owned entity which held OpenStack’s before the Foundation was formed) actually joined the OIN some time ago because of concerns about the “CPTN transaction”. The OpenStack Foundation could quite easily take over this agreement and, by virtue of the fact that the OpenStack Foundation is in some sense the entity licensing the OpenStack software to everyone, the benefits of the OIN cross-license would flow to all OpenStack users.

For OIN non-members, the Foundation could create a separate, limited license agreement whereby the OpenStack related patents held by such licensors would be licensed to the Foundation and the benefits of that agreement would also flow to all OpenStack users.

Van went on to talk a little about patent trolls (a subject close to his heart), how this is a serious issue for OpenStack and how, even now, there is a troll going after Rackspace and HP over a number issues including some related to OpenStack. Van feels that the OIN does not actually protect against the threat of trolls very well, because (somewhat obviously) such trolls would not be members of the OIN.

Van raised again (this has been discussed before) the idea of a patent indemnity program whereby OpenStack members (or even just the Foundation) would contribute to a fund which could be used for the legal defence (not any settlements themselves) of any OpenStack members threatened by patent trolls for the use of OpenStack. Van feels that patent trolls often go after some weaker companies, obtain a small settlement because that company feels it’s cheaper to settle than to defend and that settlement sets a problematic precedent that the troll can then build upon. An indemnity program would encourage members to fight patent trolls and avoid precedents being established which would set the entire community up for trouble.

Eileen made the point that the OIN was created because the GPL lacks a patent license. We’re not in the same position, since OpenStack is licensed under the Apache License which includes a broad patent license. She raised concerns about the idea of an indemnity program, how it’s scope could turn out to be very large and expensive and disagreed that trolls always go after the smaller players.

The conclusion was … no decisions today, more dialogue needed.

OpenStack Training

Next up, Jonathan revisited the topic of how the Foundation can encourage and regularize the market forming around OpenStack training.

This topic was also covered in the June meeting and Jonathan felt there was good support for some of the proposals but board members had concerns and questions about other aspects. Since then, Jonathan has been talking to several board members including Brian Stevens and Boris Renski and feels he can provide some clarifications and updates. The Foundation would like to move forward with two proposals – a training licensing program and a baseline certification test.

The training licensing program would replace the use by training providers of the Built For OpenStack mark. This will allow the Foundation to have a closer relationship with training providers and help raise awareness about the availability of OpenStack training. There is broad support for this proposal
and the Foundation have a draft agreement ready to circulate to training providers.

The baseline certification test proposal was the proposal that has required more discussion but it is better understood now. The idea is that the Foundation would define a common baseline set of training which all licensed OpenStack providers must include and certify against, but that the Foundation also fully expects providers to expand upon this content greatly and compete on the unique aspects of their training.

The vast majority of what would be included in this baseline training is already covered by all training courses out there. The Foundation would not seek to force content and topics on training providers but would instead seek to add new content when it sees that training providers are already covering these new topics. It is expected that the Foundation staff and others it engages will take 6-9 months to prepare this program.

Jonathan feels a baseline OpenStack certification would ensure that there is OpenStack training which globally available, not cost-prohibitive for students in regions all around the world and provides a target for people who want to become knowledge around OpenStack.

The board moved to support the Foundation staff in its efforts to create a baseline certification testing program for OpenStack. The motion was passed unanimously.

User Committee Update

Tim Bell took the floor next to raise an issue the User Committee has been discussing in recent times.

The committee currently consists of Tim, Ryan Lane and JC Martin. While the committee is doing a great job, they feel that their limited numbers (and their limited time) is constraining their ability to achieve some of their goals.

The obvious solution is to add more members to the committee, but there’s a catch. Committee members sign an NDA and have access to some sensitive data about OpenStack deployments which are obtained through surveys with the understanding that the information is confidential. Adding significantly more members could compromise the confidence that survey responders would have in the confidentiality of their responses.

Tim has been talking with Tom Fifield about what role Tom could take (as a Foundation staff member) on the committee in order to help out, without Tom having to sign the NDA and gain access to the confidential survey results. The idea is that Tom can take on some of the responsibility to act as a go-between who ensures that issues which are identified by the committee are raised as appropriate with the developer community.

In a similar vein, the committee is exploring the idea of a structure whereby only the “core” committee members need to sign the NDA and there would be sub-groups of the committee who can get involved without gaining direct access to the survey results.

What is Core

Rob Hirschfeld gave a quick update on the “what is core” discussions he is
leading. The discussion was initially limited to the board, more recently
broadened to include the TC and Rob now plans to broaden it to include the
wider community.

Rob mentioned a flowchart he has prepared to describe some of the proposed process. He also discussed how the initial position on requiring “plugins” has evolved into talk of designating some parts of projects as “required implementations” and allowing other parts to have “alternate implementations”.

Other Updates

Todd and Rob met with the Foundation’s legal team to discuss what changes to the Individual Directors election process would be allowed under Delaware law. He will present some thoughts on this at the next board meeting.

Simon has been working with Mark Collier and Heidi Bretz to ensure that companies who have expressed an interest in applying to be a Gold Member are aware of the board’s revised expectations around new members. Simon and the membership committee still plan on re-drafting the wiki page describing the criteria for new members.

Alan mentioned that the Compensation Committee completed a mid-year review of Jonathan’s goals for the year and concluded things are very much on track. Full details were posted to the foundation-board mailing list.

The next board meeting is scheduled for October 3rd at 9am PST. There will also be an all day board meeting in Hong Kong on November 4th.

Python Exception Chaining

OpenStack has an awful lot of developers writing Python code and many of us wouldn’t consider ourselves true “pythonistas”. This means we wind up having a bunch of interesting discussions about e.g. EABL vs LBYL.

A particular bugbear of mine is exception handling. I’m convinced that very, very few developers think hard about their error handling strategy and that the problem is even more serious with exception based languages.

So, I really like getting into error handling discussions. This morning, we have a great one – how to do exception chaining in Python, and whether that’s even something you want to do.

June 27th OpenStack Foundation Board Meeting

On June 27, the OpenStack Foundation Board of Directors met for two hours via conference call. As usual, the agenda was published in advance and the meeting was open to anyone who wanted to observe the proceedings.

These notes are my perspective of the meeting. Jonathan Bryce published an official summary and official minutes will be posted in due course.

Roll Call and Meeting Confidentiality Policy

As you can imagine, a conference call with 20-30 attendees takes a while to get going. We began with a roll call and, after perhaps 15 minutes, were ready to get started.

First item on the agenda was a review of our meeting confidentiality policy. Directors agree to refrain from commenting on the meeting proceedings until Jonathan posts his summary. The only official record of the meeting is Jonathan’s summary and the official minutes. Anything discussed during executive session is confidential. Nothing new here.

Amendment to our Certificate of Incorporation

Next up was a motion to approve a relatively minor amendment to the foundation’s certificate of incorporation.

The details of the amendment is fairly obscure, but essentially the Foundation is applying for U.S. 501(c) status which means it will be a tax-exempt, non-profit organization. There are various different organization types, but the
two most relevant are 501(c)(3) and 501(c)(6). OpenStack is filing for 501(c)(6) status.

Jonathan explained that, while preparing for this filing, it was noted that the original certificate of incorporation only allows (on winding up of the foundation) the foundation’s assets to be transferred to a 501(c)(3) organization. This amendment simply allows for the possibility of transferring assets to 501(c)(6) organizations.

There was some brief discussion clarifying exactly which status we were filing for and the motion was passed unanimously.

Transparency Policy

Next up, Lauren Sell explained the context for a formal transparency policy document which had been circulated to the board members and would require further discussion before being approved.

Lauren reminded us that at our meeting in April, the Transparency Working Group presented the principles for a transparency policy. Lauren had since worked with legal counsel to draft a more formal policy based on those original principles.

The main question Lauren felt needed to be cleared up was the issue of document management. The board has (and will continue to have) documents which must remain confidential. At our April meeting we had agreed that the OpenStack Infrastructure team would investigate hosting an OwnCloud instance which would act as our document store. While this was still on the team’s to-do list, it had not been prioritized over other work.

I suggested that, in the team time, we create a new mailing list for the the board (e.g. foundation-directors?) which would be open to everyone for read-only subscription and we would use the current mailing list to share confidential documents. Once the document management system is in place, we could then shut down the private foundation-board mailing list.

Rather than discuss in any great detail, it was agreed the Lauren would start a discussion on the foundation mailing list.

Summits & Marketing

Next up, Mark Collier and Lauren gave us an update on the Marketing front and how summit planning is progressing.

Lauren first talked through some excellent slides with a tonne of details about the Icehouse Design Summit in Hong Kong from Nov 5-8, 2013. This is the first time that the summit will be held at an “international venue” (an amusing term if you’re not U.S. based 🙂 and we again expect record attendance.

Included in Lauren’s slides were some really helpful maps and aerial shots showing the venue, the geography of Hong Kong and the location of the recommended hotels. The venue is located near the airport which is a 25 minute train journey from down town Hong Kong. There are a couple of hotels adjacent to the venue and most of the other recommended hotels are down town. The foundation staff have worked hard to come up with a good range of hotel options, including hotels with a rate of under $150 per night.

In terms of travel advice, it was noted that visitors must have a passport valid for at least one month after their planned stay and that flights from SFO to HKG are currently averaging between $1000 and $1400. Jonathan recommends that people book their flights early, because fares will increase very significantly closer to the event. Lauren also pointed out that it’s sensible to make hotel books now, since the hotels closest to the venue are already selling out.

Lauren then talked through the planned format for the summit, which has been heavily influenced by feedback received through the survey results from the previous summit.

This time around there will be two types of passes. A more affordable limited access pass will give access to the expo hall, general sessions and a single track of breakout sessions on Tuesday and Wednesday. The hope is that this will help control the numbers at the breakout sessions, but also make the event more accessible to folks who just want to come along for the first time and learn about OpenStack.

The primary language of the event will be English, but there will be simultaneous translation into Mandarin in the main hall.

The call for sponsors is already open and we have 21 sponsors to date. The headline sponsorship sold out in an astonishing 7 minutes.

For the first time, there will be a travel support program designed to ensure that lack funding won’t prevent key technical contributors from attending the event. Details of this will be announced very soon. We had a brief discussion about how this program should be run and it was pointed out that we could learn from similar programs for PyCon and UDS.

In terms of learnings from the previous summit, some of the things the team will be working hard to improve is the quality of network connectivity, the size breakout rooms and the variety of beverages and snacks.

It was noted that feedback from ATCs who completed the survey was 2.5:1 in favour of keeping the design summit collocated with the conference. In Hong Kong, the design summit rooms will be well separate from the breakout session rooms, ATC status will be properly indicated on name badges and it will be much more clear on the schedule which sessions are part of the design summit and which are breakout sessions.

After some interesting discussion about the plans for Hong Kong, Lauren gave a brief overview of how plans are proceeding for the 2014 summits. The spring summit is planned for the week of May 14 with Atlanta and Chicago under consideration. The autumn/fall summit will be one of the first two weeks of November with Paris and Berlin currently under consideration. Decisions on the venues for both these summits are expected to be made soon.

Finally, Lauren ran through some updates on the progress of the marketing community more generally. Version 1 of the OpenStack marketing portal has been made available. The mailing list is gaining new subscribers all the time. The monthly meetings are also seeing growing numbers attending.

Patent Cooperation Proposal

Next on the agenda was a presentation from several members of the Legal Affairs Committee on the three options they recommend the foundation should consider for increased cooperation on patent sharing or cross-licensing between foundation members.

Frankly, I don’t really have the energy to try and summarise these proposals in great detail, so this is short … however, this is certainly a complex and important topic.

The Apache License v2.0 has a patent provision which means you grant a license to any of your patents which are infringed upon by any contributions you make. If any licensee of the software claims that the software infringes on their patent, then they lose any patent rights granted to them under the license.

Two options were presented to the board for how we might encourage further sharing of patents related to OpenStack between the companies involved. The idea is that we could put OpenStack in a better defensive position by sharing a wider range of applicable patents.

The first option proposed was to closely copy Google’s Open Patent Non-Assertion Pledge. The idea is that companies involved in OpenStack would pledge to not assert specific sets of relevant patents against OpenStack users.

The alternative option proposed was the adoption of an OIN-style patent cross-licensing scheme. The primary difference of this scheme is that an actual patent license is granted to users, rather than just a non-assertion pledge.

The slides outlining these options will be posed to the foundation wiki page. It is hoped the board will be in a position to come to a decision on this in November.

Closing Topics

Alice King is stepping down from her role on the Legal Affairs Committee, so the board voted to approve a motion to appoint Van Lindberg to the committee.

Rob Hirschfeld gave an update on his work to bring about a productive discussion on the question of “what is core?”. Rob has held a couple of meetings with other board members and drafted six position statements which he hopes will drive the discussion towards a consensus-based decision. Rob wishes the board to come to a good level of consensus on these position statements before opening the discussion up to the rest of the community.

Finally, there was a brief discussion about training and how members of the User Committee are actively working on training materials.

May 30th OpenStack Foundation Board Meeting

Last Thursday, on May 30, the OpenStack Foundation Board met over the phone for two hours to discuss a number of topics. The date for the meeting was set well in advance, the call was open to anyone who cared to listen in and agenda was posted in advance to our wiki page.

Below is my summary of our discussions. The official summary was posted earlier by Jonathan Bryce.

For this meeting, Lew Tucker acted as chairman in place of Alan Clark.

Training

The first big topic on the agenda was an update from Mark Collier on the Foundation’s work-in-progress plans for official OpenStack training and certification programs.

The idea is that, with the huge interest globally in OpenStack, we’re hearing consistently that there is a shortage of people with OpenStack expertise and training. The Foundation would like to address this in a way that adds new Openstack experts, grows our community and establishes a base knowledge set that everyone is united around.

The OpenStack ecosystem is already ramping up its training offerings and classes are available today from a number of companies all over the world. The Foundation wants to encourage this and help accelerate the availability of training.

Crucially, the Foundation proposes to introduce a new trademark program which would require all OpenStack training and credentialling providers to include a base set of training material and tests which would be developed by the Foundation itself. The hope is that we would protect the OpenStack brand by ensuring that all official training courses would have the same basic content and quality levels.

This proposal of a new trademark program triggered a significant amount of debate which continued on over email after the call. On the call, Jim Curry and Boris Renski kicked off the discussion by expressing similar concerns about whether the trademark program would actually hinder the growth of training offerings in the ecosystem. Jonathan clarified that intent isn’t to prevent training programs from competing with additional content, but rather to ensure all programs have a common baseline. Others like Nick Barcet and Todd Moore chimed in with the view that OpenStack really needs this and Nick drew an analogy with Linux Professional Institute certification.

After much discussion, the conclusion was that the topic needed further discussion before any concrete steps could be taken. Mark Collier closed out the topic by making the point that the “Built for OpenStack” trademark was currently being used for OpenStack training and this would continue until an alternative plan was put in place.

Expect to see plenty more discussion about this soon on the foundation mailing list!

Next Board Meeting

We had some discussion about when our next meeting should be held and, based on the availability of board members, it looks like it will be held on Thursday, June 27 between 9am and 11am Pacific time.

Gold Member Committee

Next up, Simon Anderson discussed a new committee that we agreed to set up at our previous meeting – the Prospective Gold Member Committee.

The idea with this committee is that when companies approach the Foundation and express an interest in becoming a Gold Member, the Committee will work with the Foundation staff and the prospective member to ensure their application is properly prepared before it comes before the board.

The committee will essentially act as a mentor for prospective new members and help them understand what is expected of Gold Members. The hope is that this will result in applicants being better prepared than we’ve seen previously.

One concern expressed by Lauren Sell and others is that the committee shouldn’t become a vetting committee. The committee doesn’t have the mandate to turn away unsuitable candidates. If a candidate chooses to ignore the committee’s advice and mentorship, they should still be able to have their application heard by the board even this ultimately means the application is likely to be received unfavourably by the board. This point was accepted and everyone was in agreement on the mandate for the committee.

The members of the committee will be Simon Anderson, Devin Carlen, Rob Hirschfeld, Joseph George, Sean Roberts and Nick Barcet.

Update from the Executive Director

Next, Jonathan gave the board a quick update.

Jonathan talked about his attending a number of OpenStack events internationally recently and how there is still tremendous opportunity to grow the OpenStack community and engage new people. He also talked about how he met with a number of significant OpenStack users and hopes to be able to use their stories to illustrate the truly global nature of the OpenStack user community.

He also talked about the success of the OpenStack Summit in Portland and how we had 2600 attendees compared to the 1300 attendees six months previously. Feedback on the summit has been overwhelmingly positive with the most common negative comments related to the Wi-Fi network and how some of the breakout sessions were massively over-subscribed. The Foundation will continue to invest significantly in networking infrastructure at the conference and there is work underway to restructure some of the room layouts at the upcoming Summit in Hong Kong based on the feedback from Portland.

Jonathan also talked about how our financials are in good shape. We had a surplus from the Portland Summit which will allow us to make the Hong Kong Summit a kick-ass event. The Foundation is also well advanced in completing its first audit and we expect to see fully audited financials for 2012 published in August. Apparently this will be a good milestone in our progress towards non-profit status.

Role of Core

After Jonathan’s update, board members had an opportunity to briefly raise topics of interest.

First of those was Rob Hirschfeld who wants to bring together a group of board members to discuss the what “Core” project status means and should mean in future.

Rob talked about how he feels that the issue of whether core projects need a plugin architecture will be the key to unlocking the discussion and making progress.

Working With Standards Bodies

Randy Bias mentioned that he had been approached by someone from the IEEE who talked about the possibility of the IEEE working together with OpenStack on interoperability and standardization issues. Randy was mostly just passing on the message but also made the point that we can probably use the the experience of other bodies to help us ensure interoperability between OpenStack clouds.

Joshua McKenty quickly took a firm and contrary view to Randy’s – that standardization bodies are typically pretty ineffective and would actually slow down our progress on OpenStack interoperability.

The discussion concluded with general agreement that while individuals are welcome to talk to and learn from whoever they wish as part of their efforts to help make progress on OpenStack interoperability. Josh also agreed to provide the board with an update on his work on refstack.

Meeting Summaries

The final brief topic was a joy indeed. As had already been discussed on the mailing list, Josh felt that my previous meeting summary breached a policy agreed by the board (before my joining) that directors would make no public comment board meetings until after Jonathan had published an official summary of the meeting. Josh also felt that my reference to the agenda of the executive session had breached the confidentiality of the session.

Jonathan and I repeated the point that Jonathan had not gotten around to doing an official summary in a reasonable time and explicitly given me the go-ahead to post my summary. In some follow-up emails, I also made the point that – in this case – we actually made no effort to keep the agenda of the executive session private during the public part of the meeting so it was completely appropriate to mention it in my summary.

Async I/O and Python

When you’re working on OpenStack, you’ll probably hear a lot of references to ‘async I/O’ and how eventlet is the library we use for this in OpenStack.

But, well … what exactly is this mysterious ‘asynchronous I/O’ thing?

The first thing to think about is what happens when a process calls a system call like write(). If there’s room in the write buffer, then the data gets copied into kernel space and the system call returns immediately.

But if there isn’t room in the write buffer, what happens then? The default behaviour is that the kernel will put the process to sleep until there is room available. In the case of sockets and pipes, space in the buffer usually becomes available when the other side reads the data you’ve sent.

The trouble with this is that we usually would prefer the process to be doing something useful while waiting for space to become available, rather than just sleeping. Maybe this is an API server and there are new connections waiting to be accepted. How can we process those new connections rather than sleeping?

One answer is to use multiple threads or processes – maybe it doesn’t matter if a single thread or process is blocked on some I/O if you have lots of other threads or processes doing work in parallel.

But, actually, the most common answer is to use non-blocking I/O operations. The idea is that rather than having the kernel put the process to sleep when no space is available in the write buffer, the kernel should just return a “try again later” error. We then using the select() system call to find out when space has become available and the file is writable again.

Below are a number of examples of how to implement a non-blocking write. For each example, you can run a simple socket server on a remote machine to test against:

$> ssh -L 1234:localhost:1234 some.remote.host 'ncat -l 1234 | dd of=/dev/null'

The way this works is that the client connects to port 1234 on the local machine, the connection is forwarded over SSH to port 1234 on some.remote.host where ncat reads the input, writes the output over a pipe to dd which, in turn, writes the output to /dev/null. I use dd to give us some information about how much data was received when the connection closes. Using a distant some.remote.host will help illustrate the blocking behaviour because data clearly can’t be transferred as quickly as the client can copy it into the kernel.

Blocking I/O

To start with, let’s look at the example of using straightforward blocking I/O:

import socket

sock = socket.socket()
sock.connect(('localhost', 1234))
sock.send('foo\n' * 10 * 1024 * 1024)

This is really nice and straightforward, but the point is that this process will spend a tonne of time sleeping while the send() method completes transferring all of the data.

Non-Blocking I/O

In order to avoid this blocking behaviour, we can set the socket to non-blocking and use select() to find out when the socket is writable:

import errno
import select
import socket

sock = socket.socket()
sock.connect(('localhost', 1234))
sock.setblocking(0)

buf = buffer('foo\n' * 10 * 1024 * 1024)
print "starting"
while len(buf):
    try:
        buf = buf[sock.send(buf):]
    except socket.error, e:
        if e.errno != errno.EAGAIN:
            raise e
        print "blocking with", len(buf), "remaining"
        select.select([], [sock], [])
        print "unblocked"
print "finished"

As you can see, when send() returns an EAGAIN error, we call select() and will sleep until the socket is writable. This is a basic example of an event loop. It’s obviously a loop, but the “event” part refers to our waiting on the “socket is writable” event.

This example doesn’t look terribly useful because we’re still spending the same amount of time sleeping but we could in fact be doing useful rather than sleeping in select(). For example, if we had a listening socket, we could also pass it to select() and select() would tell us when a new connection is available. That way we could easily alternate between handling new connections and writing data to our socket.

To prove this “do something useful while we’re waiting” idea, how about we add a little busy loop to the I/O loop:

        if e.errno != errno.EAGAIN:
            raise e

        i = 0
        while i < 5000000:
            i += 1

        print "blocking with", len(buf), "remaining"
        select.select([], [sock], [], 0)
        print "unblocked"

The difference is we’ve passed a timeout of zero to select() – this means select() never actually block – and any time send() would have blocked, we do a bunch of computation in user-space. If we run this using the ‘time’ command you’ll see something like:

$> time python ./test-nonblocking-write.py 
starting
blocking with 8028160 remaining
unblocked
blocking with 5259264 remaining
unblocked
blocking with 4456448 remaining
unblocked
blocking with 3915776 remaining
unblocked
blocking with 3768320 remaining
unblocked
blocking with 3768320 remaining
unblocked
blocking with 3670016 remaining
unblocked
blocking with 3670016 remaining
...
real    0m10.901s
user    0m10.465s
sys     0m0.016s

The fact that there’s very little difference between the ‘real’ and ‘user’ times means we spent very little time sleeping. We can also see that sometimes we get to run the busy loop multiple times while waiting for the socket to become writable.

Eventlet

Ok, so how about eventlet? Presumably eventlet makes it a lot easier to implement non-blocking I/O than the above example? Here’s what it looks like with eventlet:

from eventlet.green import socket

sock = socket.socket()
sock.connect(('localhost', 1234))
sock.send('foo\n' * 10 * 1024 * 1024)

Yes, that does look very like the first example. What has happened here is that by creating the socket using eventlet.green.socket.socket() we have put the socket into non-blocking mode and when the write to the socket blocks, eventlet will schedule any other work that might be pending. Hitting Ctrl-C while this
is running is actually pretty instructive:

$> python test-eventlet-write.py 
^CTraceback (most recent call last):
  File "test-eventlet-write.py", line 6, in 
    sock.send('foo\n' * 10 * 1024 * 1024)
  File ".../eventlet/greenio.py", line 289, in send
    timeout_exc=socket.timeout("timed out"))
  File ".../eventlet/hubs/__init__.py", line 121, in trampoline
    return hub.switch()
  File ".../eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
  File ".../eventlet/hubs/hub.py", line 236, in run
    self.wait(sleep_time)
  File ".../eventlet/hubs/poll.py", line 84, in wait
    presult = self.do_poll(seconds)
  File ".../eventlet/hubs/epolls.py", line 61, in do_poll
    return self.poll.poll(seconds)
KeyboardInterrupt

Yes, indeed, there’s a whole lot going on behind that innocuous looking send() call. You see mention of a ‘hub’ which is eventlet’s name for an event loop. You also see this trampoline() call which means “put the current code to sleep until the socket is writable”. And, there at the very end, we’re still sleeping in a call to poll() which is basically the same thing as select().

To show the example of doing some “useful” work rather than sleeping all the time we run a busy loop greenthread:

import eventlet
from eventlet.green import socket

def busy_loop():
    while True:
        i = 0
        while i < 5000000:
            i += 1
        print "yielding"
        eventlet.sleep()
eventlet.spawn(busy_loop)

sock = socket.socket()
sock.connect(('localhost', 1234))
sock.send('foo\n' * 10 * 1024 * 1024)

Now every time the socket isn’t writable, we switch to the busy_loop() greenthread and do some work. Greenthreads must cooperatively yield to one another so we call eventlet.sleep() in busy_loop() to once again poll the socket to see if its writable. Again, if we use the ‘time’ command to run this:

$> time python ./test-eventlet-write.py 
yielding
yielding
yielding
...
real    0m5.386s
user    0m5.081s
sys     0m0.088s

you can see we’re spending very little time sleeping.

(As an aside, I was going to take a look at gevent, but it doesn’t seem fundamentally different from eventlet. Am I wrong?)

Twisted

Long, long ago, in times of old, Nova switched from twisted to eventlet so it makes sense to take a quick look at twisted:

from twisted.internet import protocol
from twisted.internet import reactor

class Test(protocol.Protocol):
    def connectionMade(self):
        self.transport.write('foo\n' * 2 * 1024 * 1024)

class TestClientFactory(protocol.ClientFactory):
    def buildProtocol(self, addr):
        return Test()

reactor.connectTCP('localhost', 1234, TestClientFactory())
reactor.run()

What complicates the example most is twisted protocol abstraction which we need to use simply to write to the socket. The ‘reactor’ abstraction is simply twisted’s name for an event loop. So, we create a on-blocking socket, block in the event loop (using e.g. select()) until the connection completes and then
write to the socket. The transport.write() call will actually queue a writer in the reactor, return immediately and whenever the socket is writable, the writer will continue its work.

To show how you can run something in parallel, here’s how to run some code in a deferred callback:

def busy_loop():
    i = 0
    while i < 5000000:
        i += 1
    reactor.callLater(0, busy_loop)

reactor.connectTCP(...)
reactor.callLater(0, busy_loop)
reactor.run()

I’m using a timeout of zero here and it shows up a weakness in both twisted and eventlet – we want this busy_loop() code to only run when the socket isn’t writeable. In other words, we want the task to have a lower priority than the writer task. In both twisted and eventlet, the timed tasks are run before the
I/O tasks and there is no way to add a task which is only run if there are no runnable I/O tasks.

GLib

My introduction to async I/O was back when I was working on GNOME (beginning with GNOME’s CORBA ORB, called ORBit) so I can’t help comparing the above abstractions to GLib’s main loop. Here’s some equivalent code:

/* build with gcc -g -O0 -Wall $(pkg-config --libs --cflags glib-2.0) test-glib-write.c -o test-glib-write */

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#include <glib.h>

GMainLoop    *main_loop = NULL;
static gchar *strv[10 * 1024 * 1024];
static gchar *data = NULL;
int           remaining = -1;

static gboolean
socket_writable(GIOChannel   *source,
                GIOCondition  condition,
                gpointer      user_data)
{
  int fd, sent;

  fd = g_io_channel_unix_get_fd(source);
  do
    {
      sent = write(fd, data, remaining);
      if (sent == -1)
        {
          if (errno != EAGAIN)
            {
              fprintf(stderr, "Write error: %s\n", strerror(errno));
              goto finished;
            }
          return TRUE;
        }

      data = &data[sent];
      remaining -= sent;
    }
  while (sent > 0 && remaining > 0);

  if (remaining <= 0)
    goto finished;

  return TRUE;

 finished:
  g_main_loop_quit(main_loop);
  return FALSE;
}

static gboolean
busy_loop(gpointer data)
{
  int i = 0;
  while (i < 5000000)
    i += 1;
  return TRUE;
}

int
main(int argc, char **argv)
{
  GIOChannel         *io_channel;
  guint               io_watch;
  int                 fd;
  struct sockaddr_in  addr;
  int                 i;
  gchar              *to_free;

  for (i = 0; i < G_N_ELEMENTS(strv)-1; i++)
    strv[i] = "foo\n";
  strv[G_N_ELEMENTS(strv)-1] = NULL;

  data = to_free = g_strjoinv(NULL, strv);
  remaining = strlen(data);

  fd = socket(AF_INET, SOCK_STREAM, 0);

  memset(&addr, 0, sizeof(struct sockaddr_in));
  addr.sin_family      = AF_INET;
  addr.sin_port        = htons(1234);
  addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);

  if (connect(fd, (struct sockaddr *)&addr, sizeof(addr)) == -1)
    {
      fprintf(stderr, "Error connecting to server: %s\n", strerror(errno));
      return 1;
    }

  fcntl(fd, F_SETFL, O_NONBLOCK);

  io_channel = g_io_channel_unix_new(fd);
  io_watch = g_io_add_watch(io_channel,
                            G_IO_OUT,
                            (GIOFunc)socket_writable,
                            GINT_TO_POINTER(fd));

  g_idle_add(busy_loop, NULL);

  main_loop = g_main_loop_new(NULL, FALSE);

  g_main_loop_run(main_loop);
  g_main_loop_unref(main_loop);

  g_source_remove(io_watch);
  g_io_channel_unref(io_channel);

  close(fd);

  g_free(to_free);

  return 0;
}

Here I create a non-blocking socket, set up an ‘I/O watch’ to tell me when the socket is writable and, when it is, I keep blasting data into the socket until I get an EAGAIN. This is the point at which write() would block if it was a blocking socket and I return TRUE from the callback to say “call me again when the socket is writable”. Only when I’ve finished writing all of the data do I return FALSE and quit the main loop causing the g_main_loop_run() call to return.

The point about task priorities is illustrated nicely here. GLib does have the concept of priorities and has a “idle callback” facility you can use to run some code when no higher priority task is waiting to run. In this case, the busy_loop() function will *only* run when the socket is not writable.

Tulip

There’s a lot of talk lately about Guido’s Asynchronous IO Support Rebooted (PEP3156) efforts so, of course, we’ve got to have a look at that.

One interesting aspect of this effort is that it aims to support both the coroutine and callbacks style programming models. We’ll try out both models below.

Tulip, of course, has an event loop, time-based callbacks, I/O callbacks and I/O helper functions. We can build a simple variant of our non-blocking I/O example above using tulip’s event loop and I/O callback:

import errno
import select
import socket

import tulip

sock = socket.socket()
sock.connect(('localhost', 1234))
sock.setblocking(0)

buf = memoryview(str.encode('foo\n' * 2 * 1024 * 1024))
def do_write():
    global buf
    while True:
        try:
            buf = buf[sock.send(buf):]
        except socket.error as e:
            if e.errno != errno.EAGAIN:
                raise e
            return

def busy_loop():
    i = 0
    while i < 5000000:
        i += 1
    event_loop.call_soon(busy_loop)

event_loop = tulip.get_event_loop()
event_loop.add_writer(sock, do_write)
event_loop.call_soon(busy_loop)
event_loop.run_forever()

We can go a step further and use tulip’s Protocol abstraction and connection helper:

import errno
import select
import socket

import tulip

class Protocol(tulip.Protocol):

    buf = b'foo\n' * 10 * 1024 * 1024

    def connection_made(self, transport):
        event_loop.call_soon(busy_loop)
        transport.write(self.buf)
        transport.close()

    def connection_lost(self, exc):
        event_loop.stop()
 
def busy_loop():
    i = 0
    while i < 5000000:
        i += 1
    event_loop.call_soon(busy_loop)

event_loop = tulip.get_event_loop()
tulip.Task(event_loop.create_connection(Protocol, 'localhost', 1234))
event_loop.run_forever()

This is pretty similar to the twisted example and shows up yet another example of the lack of task prioritization being an issue. If we added the busy loop to the event loop before the connection completed, the scheduler would run the busy loop every time the connection task yields.

Coroutines, Generators and Subgenerators

Under the hood, tulip depends heavily on generators to implement coroutines. It’s worth digging into that concept a bit to understand what’s going on.

Firstly, remind yourself how a generator works:

def gen():
    i = 0
    while i < 2:
        print(i)
        yield
        i += 1

i = gen()
print("yo!")
next(i)
print("hello!")
next(i)
print("bye!")
try:
    next(i)
except StopIteration:
    print("stopped")

This will print:

yo!
0
hello!
1
bye!
stopped

Now imagine a generator function which writes to a non-blocking socket and calls yield every time the write would block. You have the beginnings of coroutine based async I/O. To flesh out the idea, here’s our familiar example with some generator based infrastructure around it:

import collections
import errno
import select
import socket

sock = socket.socket()
sock.connect(('localhost', 1234))
sock.setblocking(0)

def busy_loop():
    while True:
        i = 0
        while i < 5000000:
            i += 1
        yield

def write():
    buf = memoryview(b'foo\n' * 2 * 1024 * 1024)
    while len(buf):
        try:
            buf = buf[sock.send(buf):]
        except socket.error as e:
            if e.errno != errno.EAGAIN:
                raise e
            yield
    quit()

Task = collections.namedtuple('Task', ['generator', 'wfd', 'idle'])

tasks = [
    Task(busy_loop(), wfd=None, idle=True),
    Task(write(), wfd=sock, idle=False)
]

running = True

def quit():
    global running
    running = False

while running:
    finished = []
    for n, t in enumerate(tasks):
        try:
            next(t.generator)
        except StopIteration:
            finished.append(n)
    map(tasks.pop, finished)

    wfds = [t.wfd for t in tasks if t.wfd]
    timeout = 0 if [t for t in tasks if t.idle] else None

    select.select([], wfds, [], timeout)

You can see how the generator-based write() and busy_loop() coroutines are cooperatively yielding to one another just like greenthreads in eventlet would do. But, there’s a pretty fundamental flaw here – if we wanted to refactor the code above to re-use that write() method to e.g. call it multiple times with
different input, we’d need to do something like:

def write_stuff():
    for i in write(b'foo' * 10 * 1024 * 1024):
        yield
    for i in write(b'bar' * 10 * 1024 * 1024):
        yield

but that’s pretty darn nasty! Well, that’s the whole idea behind Syntax for Delegating to a Subgenerator (PEP380). Since python 3.3, a generator can now yield to another generator using the ‘yield from’ syntax. This allows us to do:

...
def write(data):
    buf = memoryview(data)
    while len(buf):
        try:
            buf = buf[sock.send(buf):]
        except socket.error as e:
            if e.errno != errno.EAGAIN:
                raise e
            yield

def write_stuff():
    yield from write(b'foo\n' * 2 * 1024 * 1024)
    yield from write(b'bar\n' * 2 * 1024 * 1024)
    quit()

Task = collections.namedtuple('Task', ['generator', 'wfd', 'idle'])

tasks = [
    Task(busy_loop(), wfd=None, idle=True),
    Task(write_stuff(), wfd=sock, idle=False)
]
...

Conclusions?

Yeah, this is the point where I’ve figured out what we should do in OpenStack. Or not.

I really like the explicit nature of Tulip’s model – for each async task, you explicitly decide whether to block the current coroutine on its completion (or put another way, yield to another coroutine until the task has completed) or you register a callback to be notified of the tasks completion. I’d much prefer this to rather cavalier “don’t worry your little head” approach of hiding the async nature of what’s going on.

However, the prospect of porting something like Nova to this model is more than a little dauting. If you think about the call stack of an REST API request being handled and ultimately doing an rpc.cast() and that the entire call stack would need to be ported to ‘yield from’ in order for us to yield and handle another API request while waiting for the result of rpc.cast() …. as I said, daunting.

What I’m most interested in is how to design our new messaging API to be able to support any and all of these models in future. I haven’t quite figured that out either, but it feels pretty doable.

April 14th OpenStack Foundation Board Meeting

On April 14, the day before the OpenStack Summit in Portland began, the OpenStack Foundation Board held an all-day, in-person meeting. Like all of our board meetings, the agenda was packed solid.

After our meetings, the goal is for Jonathan to post a summary to the mailing list (like this one) and I try and follow up with a longer blog post with a bit more commentary. For a bunch of reasons, we’ve let things slide this time, but here’s my attempt at a summary anyway. It’s been a while since the meeting, so a lot of the details are pretty vague at this point.

One thing I really liked about the meeting was that we had chairs set up so that anyone who wished to attend the meeting and listen were welcome to do so. With so many stackers already in town for the summit, it really was a great opportunity to open the workings of the board to anyone interested.

Transparency Committee

The first meaty item on the agenda was an update from Joshua McKenty on the progress of the board’s Transparency Committee.

We began by discussing Nick Barcet’s efforts to choose and implement a document management system for the board. OwnCloud, Google Drive, Dropbox and github have all been considered. Google Drive and Dropbox would not be accessible by all board members. It was noted that the Foundation staff will also require a document management system and we might be able to use the same system for both use cases. Conclusion was for Nick and Monty to see if the OpenStack Infrastructure team could host OwnCloud for this purpose.

Next we discussed the transparency policy which Lauren Sell and Mark Radcliffe have been working on. The core concern is how to balance the need for having a culture of transparency and openness with the need for some information (like legal or personnel matters) to be kept confidential. The general approach discussed was having the board’s mailing list open, but confidential information would be shared via private documents in a document management system that could be referred to in mailing list posts. The distinction between confidential and embargoed information was discussed and it was agreed that embargoed information should have a disclosure date associated with it.

There was also a brief discussion on the possibility of having a transparency ombudsman. This would be a person trusted by the community with whom complaints related to transparency could be raised. It was felt this could be a duty of a foundation employee, if that individual had already established the reputation for trustworthiness and independence required.

2013 Individual Director Elections

Next up, Rob Hirschfeld led a discussion on possible changes to the election process in time for the 2013 Individual Director elections.

We began by discussing how we would go about making any required changes to the bylaws. It was recognized that we would need a firm proposal to be made to the board in August so that notice could be given in September of a vote in October in time for the elections in November.

It was also noted that a bylaws change would require a majority vote of the individual members with a voter turnout of 25%. For reference, we had a 28% turnout at the previous election.

We also briefly discussed the potential objectives for any changes to the elections process but avoided going deep on this because of the limited time available on the agenda.

Legal Affairs Committee

Next on the agenda was a discussion (led by me) about the Legal Affairs Committee.

The main concern I raise was the project (or the “Technical Community and Committee”, as I called it) needed the support of subject matter experts when dealing with the kind of legal issues regularly encountered by all open-source projects, particularly in the area of copyright and licensing. I proposed Richard Fontana’s idea that, rather than depending exclusively on the Legal Affairs Committee or the Foundation’s legal counsel for help with such matters, we would create a legal-discuss mailing list where any subject matter experts could contribute to resolving issues raise by the technical community. The idea was that we would also have a FAQ wiki page to document any conclusions reache on the list. There was some discussion about whether opinions expressed on the list could be construed as legal advice and it was agreed to put in place disclaimers to avoid that perception.

(For the record, we’ve since created the legal-discuss mailing list and the legal issues FAQ.)

We also discussed whether the description of scope of the Legal Affairs Committee in the bylaws could be improved and whether the strict five member limit in the bylaws was really such a good idea. It was agreed that I would work with Alice King to see if we can address those issues.

(Again for the record, Alice and I had a productive discussion and we put together the Legal Affairs Committee wiki page to at least ensure the committee’s scope, membership and progress could be properly documented.)

Gold Member Applications

Next up, we had presentations from Juniper and Ericsson on their applications for Gold Membership. Once the presentations had been completed, we convened an executive session to discuss the applications in private.

During the executive session, the board reviewed each application using the [https://wiki.openstack.org/wiki/Governance/Foundation/PotentialMemberCriteria previously agreed criteria].

Once the executive session was completed, the board reconvened the meeting and carried out the official, public vote on the new member applications. The applications of both Ericsson and Juniper were approved.

As part of the voting process, some directors chose to publicly restate their concerns with the applications that had been discussed during the executive session. One concern raised by Rob Hirschfeld and Joshua McKenty was that, while both applicants shared their plans for increasing their OpenStack engagement, it could be argued that future plans aren’t the same as the “demonstrated commitment” talked about in the membership criteria.

Another concern discussed at length was that applicants could provide much more in the way of written detail about how they meet the membership criteria. One take on this was that applicants should be expected to provide a resume of their involvement with, and commitment, to OpenStack. Simon Anderson proposed that we form a membership committee which would mentor applicants and advise them on how to prepare their application.

In total, discussion of these applications and the process for new gold members took up 4 hours of the board’s time. While this may seem excessive, it was very apparent that everyone on the board was keen for the Foundation to have gold members that would help drive OpenStack’s success. With the addition of Ericsson and Juniper, we now have 16 of the 24 gold member slots filled. As the number of remaining slots shrink, I think we will see the board having higher expectations of applications.

Programs

At this point, the board seemed collectively quite drained of energy and the agenda had to be significantly reworked because there wasn’t much time remaining.

Monty took the opportunity to briefly discuss an idea he had to introduce the concept of “programs” in addition to our current concept of projects. We would use this to recognize efforts like documentation and CI on equal footing with the integrated projects. Other efforts like Oslo, TripleO and puppet recipes were also mentioned as potential programs.

Rather than go into too detailed discussion on the topic, we generally expressed support for the concept but agreed it needed further discussion.

Joint Session

Part of the agenda for the meeting was that members of the Technical Committee were invited to join the meeting for a joint session. Monty and I were obviously already present, but Michael Still and Thierry Carrez also joined. Other TC members had not yet arrived in Portland or weren’t fully aware of the joint session. We had some discussion about how to have a joint session with better attendance from both groups and the idea of a joint design summit session at the next summit was mooted.

User Committee

Tim Bell stepped up next to give an overview of the results of the User Committee’s recently completed survey of OpenStack users.

Tim is presenting these exciting results at the summit, so I won’t repeat them here. However, we did have an interesting discussion about how representative the survey was of OpenStack deployments. Tim offered 50-75% as his gut feeling for the percentage of deployments covered but Joshua McKenty reckoned it was probably less than 25%. There was some discussion about future surveys and the potential for an opt-in usage stats tracking tool to increase the percentage of deployments we know about.

Incubation Update Committee

Next up, Alan Clark presented the final report of the Incubation Update Committee.

The board was generally very supportive of the work of the committee, but it was clear further discussion was required on the topic of Core status and the board’s trademark usage program. Much discussion was had around the desire for projects like Heat and Ceilometer to be able to adopt official OpenStack project names (like “OpenStack Measurement” for Ceilometer) and whether this was all that would be implied by Core status. However, there is also the question of whether clouds which wish to use the OpenStack brand should be required to use all Core projects or whether a trademark program around more targeted interoperability testing made more sense.

It was agreed that this was an incredibly important and urgent topic but that we had run out of time at this meeting to make progress on it.

February 12th OpenStack Foundation Board Meeting

Yesterday was the first in-person meeting of the 2013 Foundation Board. We met at Rackspace’s offices in San Francisco at Jim Curry’s invitation. The meeting also coincided with the Foundation staff all getting together in-person for the first time. With so many OpenStack people converging on the one place, it was quite funny to randomly run into various people at the Courtyard Marriott around the corner.

The meeting kicked off at 10am after some informal introductions. For me, it was a case of playing a “hey, are you Tristan?”/”No, I’m Sean” guessing game. Monty brought his usual wackiness to the ocassion by handing out beads and wearing a luminous, sparkling orange fedora with bright blue flashing LEDs. It’s Mardi Gras time!

Formalities

The first item on the agenda was a formal role call. 5 board members couldn’t attend and Tim Bell was on the phone. Also attending were Jonathan Bryce and Mark from the Foundation staff. Mark Radcliffe, our outside council, was on the phone.

From that start, it was obvious that it was going to be extremely difficult for those dialled into the meeting to be able to participate effectively. This is definitely something we will continue to struggle with and, as one of the few who had to travel halfway around the world for the meeting, I can certainly sympathise with those on the phone.

As a further formality, Alan Clark reviewed some of the existing board policies:

  • that observers are allowed at board meetings, except in the case of executive sessions
  • that board members wouldn’t blog or tweet until after an official summary of the meeting had been posted to the foundation list

There was some discussion about how executive discussions should be handled. How we can describe what the topic of any executive sessions were and perhaps also publicly hold any votes arising from those sessions.

Rob Hirschfeld proposed that our general decision making process should involve first coming to an agreement on the criteria for making the decision and then applying the criteria. The idea was that this process would avoid some circular debates and save time.

User Committee

Next up, Ryan Lane presented an update on the progress of the User Committee. He described the mandate of the committee as “fighting for the users”. The initial goals of the committee are to define their charter and a set of categories of users. Ryan encouraged the board to review and comment on the Google Doc they had prepared.

Much discussion was had about how to gather data about our users. Options included a CRM system, user polls, anonymous usage reporting tools, aggregating statistics to protect the innocent and the like. This is clearly going to be a very hot topic for the committee.

During the discussion, Josh McKenty posted a blueprint for how a opt-in tracking system might work.

We also had an interesting debate about the committee should move forward with adding more members to the committee. The committee’s own proposal for this was to be democratic and have elections for representatives from different geographies or categories of users. Quite a few board members raised concerns about this approach and asked the committee members to press forward quickly and appoint a diverse set of members with minimal bureaucracy.

Once Ryan had finished, the board expressed their thanks and support for the efforts of the committee. Ryan was asked to stay and observe the board meeting, but Ryan preferred to go and get some “real work” done. That’s the spirit!

If you want to follow the progress of the user committee, the best way is to subscribe to user-committee@lists.openstack.org mailing list. Ryan would be happy to share his slide deck if you email him.

Legal Affairs Committee

Alice King and Nissa Strottman took the floor and presented the work of the Legal Affairs Committee on the whole area of patents and risk mitigation.

Alice and Nissa first talked the board through some background on patents. They described the “risk landscape” in terms of competing companies suing and counter-suing each other for infringement (Alice had an awesome diagram of “who’s suing who in the mobile industry” which got a good laugh from everyone) but also, perhaps more importantly, the threat of Non Practicing Entities or “patent trolls”.

We talked in some detail about the Patent Grant in the Apache License and how it does a lot to mitigate the risks but doesn’t completely eliminate them. In particular, it does little to help with the threat of NPEs. An interesting debate followed about various nuances of the Apache License definitions and how they relate to OpenStack.

Alice and Nissa described approaches taken by other communities – e.g. OIN, GPLv3, Open Compute and Eclipse – and also how an additional contributor agreement could be adopted by the project to further help. Brian Stevens talked about how the OIN works and how it isn’t strictly limited to Linux. It was noted that many of the larger members of the Foundation are also members of OIN. Eileen Evans described how the adoption of a new contributor agreement would be a massive bureaucratic challenge for some of the larger companies involved in the project.

A point on one of the slides that a more robust patent policy would remove a barrier to OpenStack’s adoption triggered some discussion about whether patent risk is in fact hindering OpenStack’s adoption. The consensus seemed to be that this wasn’t seen as a huge issue and how, in fact, OpenStack is in as good a position as most any other open-source project. While it’s important for the Foundation to do due diligence on this matter, it’s also important that we don’t overstate the actual risk.

Another topic discussed was the question of “defensive publication” of ideas generated by the developer community so that they are properly documented in a way that is accessible to lawyers who wish to demonstrate prior art. Various ideas were suggested about how blueprints could form the basis for such an approach and how to do this without placing the burden on the developers.

It’s amusing the way we wind ourselves in knots over these things and generally come to the conclusion that the issues are difficult, but that things are working pretty well as they currently stand.

Finally, with limited time available, we picked up again on some of the discussion from the previous board meeting about the scope, name and makeup of the committee. Some board members seemed fine with how things currently stand while others reiterated the issues previously discussed.

Anyone interested in the legal affairs committee should talk to Alice King about how to get involved. She would be happy to share her slides with those interested.

Lunch

At this point, it was time to break for lunch. Most board members stayed for the lunch provided on-site and had productive, informal discussions while eating.

Monty, Nick Barcet and I attended a Technical Committee meeting over IRC while also eating and chatting with the rest of the board. That’s multi-taksing! Amazingly every single member of the TC attended the meeting and we voted unamimously to update the Incubation Process as recommended by the IncUp committee. Such a level of TC consensus is completely unprecedented and took a tonne of work to achieve. Kudos to Thierry for herding the cats on this.

Financial Report

Sean Roberts presented a report on the work of the Financial Oversight committee and we discussed the progress on completing the Foundation’s first financial audit and the updated financial forecast for the year.

The topic was so uncontroversial (you could even say boring) that we didn’t even take much in the way of notes on it. There was unanimous agreement that this is how it should be and we’d be doing something wrong if the topic was “interesting”. The board praised the committee and the staff for their dilligence in doing everything to make sure the Foundation’s financial affairs were beyond question.

Transparency

Josh McKenty was next up presenting his proposal for a Transparency Policy for the board and his wanting volunteers to form a committee to finalize the details.

He described the committee’s goal as:

To improve transparency and foster collaboration between the foundation members and members of the board, technical committee, user committee and other committees. Specifically, to draft statements and prototype systems changes for board review and approval.

A large part of the discussion was around trying to quantify the problem we’retrying to solve and understanding how we’d achieve closure on it. The board doesn’t want to be consumed with this indefinitely, so how will we know when it’s simply a case of “you can’t please everyone”.

We wrapped this up by gathering volunteers for the committee:

  • Nick Barcet
  • Jonathan Bryce
  • Alan Clark
  • Eileen Evans
  • Tristan Goode
  • Rob Hirschfeld
  • Kyle MacDonald
  • Joshua McKenty
  • Mark McLoughlin
  • Lauren Sell

Director’s Report

Next up, Jonathan provided an awesome update on the Foundation’s progress and plans. It’s clear an awful lot of time and thought went into this super-helpful and encouraging update.

One of Jonathan’s slides showed some mind-blowing statistics detailing the growth of the developer community, user community and ecosystem. The statistics on the number of patches merged every month were really stunning and the board praised the success of the project’s infrastructure team in enabling this level of activity.

Jonathan talked to his hiring plan and introduced the latest Foundation employees. We’re executing almost to the plan and there are two positions still to fill – another infrastructure engineer and another community manager.

There was a brief discussion of the updated budget. The summary is that slightly more money is both coming in and going out than planned, giving a positive net effect. A motion was passed to approve the new budget.

The discussion moved more to event planning and marketing matters and Lauren Sell filled the board in on a lot of the details in this area.

Planning of the Havana summit was well in hand and it was noted that the choice of venue means that costs will rise more linearly with attendance than previous summits. Lauren discussed the venue selection process for the October 2013 summit which should come to a conclusion soon. Future summits were discussed with general agreement that a two year cycle should be rougly East Coast US, Asia, West Coast US and Europe. Several board members expressed a desire to get started on the selection process for the October 2014 summit since the size of the event means that suitable locations get booked up very far in advance.

Lauren also described her efforts to bring the various marketers at dozens of OpenStack companies together with some of the same principles that drive the collaborative software development process. She has created a mailing list which already has almost 100 members and holds a regular phone meeting of the group. The goal is to have all those involved in marketing OpenStack to be highly co-ordinated and “on message”.

Finally, Lauren quickly presented an independent study she had just received on OpenStack’s marketing impact relative to other open-source projects and industry players. The results of the study were simply stunning and most of us appeared to be struggling to believe them. However, even if the statistics on OpenStack are massively inflated, we’re still hugely being successful. Lauren, Mark and Jonathan have already worked with the authors of the report to find any data that could undermine the conclusions and will continue to do so.

Jonathan and Lauren can be contacted directly for the materials they presented to the board.

Strategy Session

Jonathan moved on to kick off a strategy session where board members would have an opportunity to put their thinking hats on and brainstorm together. He teed up the discussion by presenting his thesis that OpenStack was building a “platform ecosystem” with huge potential for network effects that would result in OpenStack being the dominant cloud platform in the market place.

We then moved onto having a short exercise where we used sticky-notes to throw out our ideas in three areas – “interoperability”, “reference architectures” and “engaging users”. Once the ideas had been gathered, we split into breakout groups to discuss ideas for each of those areas.

Rather than trying to capture all of the ideas and action items from the discussions here, it’s perhaps easier to call out some highlights:

  • We’re going to explore the use of Tempest as an API compatibility testing tool which can be pointed at an OpenStack instance. The idea is that anyone who wants to license the OpenStack trademark would request a test run which would result in a scorecard. At every release, the board would update the
    results required for each OpenStack mark so that obtaining a trademark license simply becomes a matter of fixing any failing tests which are required for the desired mark.
  • We’re going to explore the definition of reference architecture “flavours” and how these reference architectures could be defined and maintained. The initial idea is that Heat templates could be used as a deployable reference architecture definitions.
  • In terms of engaging users, we decided that trystack.org and deployment tracking tools were crucial.

Election Process Committee

Todd Moore raised the question of whether allowing Foundation staff to run for the Individual Member Director elections was a waste of a board seat since staff members are so involved in the Foundation anyway. The question of potential conflicts of interest was also raised. There was no time to debate this question so the board resolved to reconstitute the Election Process Committee to consider this question and the general question of the eligibility to run and vote for these board seats.

The committee will consist of Todd as chair and the 8 individual board members.

Evening Event

We wrapped up at 6pm sharp and headed over to a nearby restaurant with all of the Foundation staff members. Far from simply being a social event, it was incredible to see 30+ focused and committed folks spend over 4 hours going from conversation to conversation about how to continue OpenStack’s success into the future. For me personally, those conversations alone were enough to justify the effort it took to get to the meeting.