OpenID Attribute Exchange

26 November, 2007 – 4:18 pm

In my previous article on OpenID 2.0, I mentioned the new Attribute Exchange extension. To me this is one of the more interesting benefits of moving to OpenID 2.0, so it deserves a more in depth look.

As mentioned previously, the extension is a way of transferring information about the user between the OpenID provider and relying party.

Why use Attribute Exchange instead of FOAF or Microformats?

Before deciding to use OpenID for information exchange, it is worth looking at whether it is necessary at all.

There are existing solutions for transferring user data such as FOAF and the hCard microformat. As the relying party already has the user’s identity URL, it’d be trivial to discover a FOAF file or hCard content there. That said, there are some disadvantages to this method:

  1. Any information published in this way is available to everyone. This might be fine for some classes of information (your name, a picture, your favourite colour), but not for others (your email address, phone number or similar).
  2. The same information is provided to all parties. Perhaps you want to provide different email addresses to work related sites.
  3. The RP needs to make an additional request for the data. If we can provide the information as part of the OpenID authentication request, it will reduce the number of round trips that need to be made. In turn, this should reduce the amount of time it takes to log the user in.

Why use Attribute Exchange instead of the Simple Registration extension?

There already exists an OpenID extension for transferring user details to the RP, in the form of the Simple Registration extension. It has already been used in the field, and works with OpenID 1.1 too.

One big downside of SREG is that it only supports a limited number of attributes. If you need to transfer more attributes, you basically have two choices:

  1. use some other extension to transfer the remaining attributes
  2. make up some new attribute names to send with SREG and hope for the best.

The main problem with (2) is that there is no way to tell between your own extensions to SREG and someone else’s which will likely create interoperability problems if when an attribute name conflict occurs. So this solution is not a good idea outside of closed systems. This leaves (1), for which Attribute Exchange is a decent choice.

What can I do with Attribute Exchange?

There are two primary operations that can be performed with the extension:

  1. fetch some attribute values
  2. store some attribute values

Both operations are performed as part of an OpenID authentication request. Among other things, this allows:

  • The OP to ask the user which requested attributes to send
  • If the OP has not stored values for the requested attributes, it could get the user to enter them in and store them for next time.
  • The OP could use a predefined policy to decide what to send the RP. One possibility would be to generate one-time email addresses specific to a particular RP.
  • For store requests, the OP can ask the user to confirm that they want to store the attributes.

Fetching Attributes

An attribute fetch request is a normal authentication request with a few additional fields:

  • openid.ax.mode: this needs to be set to “fetch_request”
  • openid.ax.required: a comma separated list of attribute aliases that the RP needs (note that this does not guarantee that the OP will return those attributes).
  • openid.ax.if_available: a comma separated list of attribute aliases that the RP would like returned if available.
  • openid.ax.type.alias: for each requested attribute alias, the URI identifying the attribute type
  • openid.ax.count.alias: the number of values the RP would like for the attribute.
  • openid.ax.update_url: a URL to send updates to (will be discussed later).

The use of URIs to identify attributes makes it trivial to define new attributes without conflicting with other people (and as with XML namespaces, the attribute aliases are arbitrary). However, the extension is only useful if the OP and RP can agree on attribute types. To help with this, there is a collection of community defined attribute types at axschema.org.

As an example, imagine a web log that uses OpenID to authenticate comment posts. Rather than just printing the OpenID URL for the commenter, it could use attribute exchange to request their name, email, website and hackergotchi. The authentication request might contain the following additional fields:

openid.ns.ax=http://openid.net/srv/ax/1.0
openid.ax.mode=fetch_request
openid.ax.required=name,hackergotchi
openid.ax.if_available=email,web
openid.ax.type.name=http://axschema.org/namePerson
openid.ax.type.email=http://axschema.org/contact/email
openid.ax.type.hackergotchi=http://axschema.org/media/image/default
openid.ax.type.web=http://axschema.org/contact/web/default

In the successful authentication response, the following fields will be included (assuming the OP supports the extension):

  • openid.ax.mode: must be “fetch_response”
  • openid.ax.type.alias: specify the type URI for each attribute being returned.
  • openid.ax.count.alias: the number of values being returned for the given attribute alias (defaults to 1).
  • openid.ax.value.alias: the value for the given attribute alias, if no corresponding openid.ax.count.alias field was sent.
  • openid.ax.value.alias.n: the nth value for the given attribute alias, if a corresponding openid.ax.count.alias field was sent. The first attribute value is sent with n = 1.
  • openid.ax.update_url: to be discussed later.

For the web log example given above, the response might look like:

openid.ns.ax=http://openid.net/srv/ax/1.0
openid.ax.mode=fetch_response
openid.ax.type.name=http://axschema.org/namePerson
openid.ax.type.email=http://axschema.org/contact/email
openid.ax.type.hackergotchi=http://axschema.org/media/image/default
openid.ax.value.name=John Doe
openid.ax.value.email=john@example.com
openid.ax.count.hackergotchi=0

In this response, we can see the following:

  1. The user has provided their name and email
  2. They have not provided any information about their web site. Either the OP does not support the attribute or the user has declined to provide it.
  3. The use has explicitly stated that they have no hackergotchi (i.e. it is a zero-valued attribute).

Storing Attributes

Using the Attribute Exchange fetch request, it is possible to outsource management of pretty much all the user’s profile information to the OP. That said, the user will still need to update their profile data occasionally. Telling them to go to their OP to change things and then log in again is not particularly user friendly though.

Using the store request, the RP can let the user update their profile on site and then transfer the changes back to the OP. Like the fetch request, a store request is performed as part of an OpenID authentication request. The additional request fields are pretty much identical to a store response, except that openid.ax.mode is set to “store_request”.

In the positive authentication response, the RP can see whether the data was successfully stored by checking the openid.ax.mode response field. If the data was stored, then it will be set to “store_response_success”. If the data was not stored it will be set to “store_response_failure” and an error message may be found in openid.ax.error.

Asynchronous Attribute Updates

One downside of the Simple Registration extension is that it only transferred user details on login. This means that it is only possible to get updates to attribute values by asking the user to log in again. The Attribute Exchange extension provides a way to solve this problem in the form of the openid.ax.update_url request field.

When a “fetch_request” is issued with the openid.ax.update_url field set, a compliant OP will record the following:

  1. the claimed ID and local ID from the authentication request
  2. the list of requested attributes
  3. the update_url value (after verifying that it matches the openid.realm value of the authentication request).

The OP will then include openid.ax.update_url in the authentication response as an acknowledgement to the RP. When any of the given attributes are updated the OP will send an unsolicited positive authentication response to the given update URL. This will effectively be the same as the original authentication response (i.e. for the same claimed ID and local ID), but with new values for the changed attributes.

As there is no mention of unsolicited authentication responses in the main OpenID authentication specification, it is worth looking at what checking the RP should do. This includes:

  • Is this OP still authoritative for the claimed ID? This is checked by performing discovery on the claimed ID and verifying that it results in the same server URL and local ID as given in the response.
  • Did the message come from the OP? As with a standard response, there should be a signature for the fields. Since the OP does not know what association to use for the signature, a new private association will be used. By issuing a “check_authentication” request to the OP, the RP can verify that the message originated from the OP.

If these checks fail the RP should respond with a 404 HTTP error code, which tells the OP to stop sending updates. If the message is valid, the RP can update the user’s profile data.

Caveats

While the Attribute Exchange extension provides significant features above those provided by Simple Registration, but it still has its limitations:

  1. Any attribute values provided to the RP are self-asserted.
  2. Related to the above, there is no way for a third party to make assertions about attribute values.

For (1), the solution is to perform the same level of verification on the attribute value as if the user had entered it directly. So an OpenID enabled mailing list manager should verify the email address provided by attribute exchange before subscribing the user. In contrast, an OpenID enabled shop probably doesn’t need to do further verification of the user’s shipping address (since it is in the user’s best interest to provide correct information).

The exception to this rule is when there is some other trust relationship between the OP and RP. For instance, if the RP knows that the OP will only send an email address if it has first been validated, then it may decide to trust the email address without performing its own validation checks. This is most likely to be useful in closed systems that happen to be using OpenID for single sign-on.

By James Henstridge | Posted in Uncategorised | Tagged , | Comments (4)

Weird GNOME Power Manager error message

19 November, 2007 – 7:08 am

Since upgrading to Ubuntu Gutsy I’ve occasionally been seeing the following notification from GNOME Power Manager:

GNOME Power Manager notification

I’d usually trigger this error by unplugging the AC adapter and then picking suspend from GPM’s left click menu.

My first thought on seeing this was “What’s a policy timeout, and why is it not valid?” followed by “I don’t remember setting a policy timeout”. Looking at bug 492132 I found a pointer to the policy_suppression_timeout gconf value, whose description gives a bit more information.

Apparently the timeout is designed to ignore spurious messages from the hardware after a resume — you wouldn’t want to process a left over “suspend” message immediately after resuming from suspend after all. This does bring up a few questions though:

  1. While ignoring “please suspend” messages shortly after performing a suspend makes sense, why ignore “please suspend” messages after an “on battery power message”?
  2. While messages from the hardware might be spurious, surely picking an option from GPM’s menu is not. I guess such suspend requests are being mixed in with hardware suspend requests before the point where the policy timeout is checked.
By James Henstridge | Posted in Uncategorised | Tagged , , | Comments (2)

Identifier Reuse in OpenID 2.0

11 November, 2007 – 10:15 pm

One of the issues that the OpenID 1.1 specification did not cover is the fact that an identity URL may not remain the property of a user over time. For large OpenID providers there are two cases they may run into:

  1. A user with a popular user name stops using the service, and they want to make that name available to new users.
  2. A user changes their user name. This may be followed by someone taking over the old name.

In both cases, RPs would like some way to tell the difference between two different users who present the same ID at different points in time.

The traditional method of solving this problem is to assign two identifiers to a user: a human friendly identifier and a persistent identifier (e.g. a UNIX user ID, a database row ID, etc). At any point in time, the human friendly identifier will point to a particular persistent identifier, but over time the relationship may not hold. Whenever a human-friendly identifier is presented, it is transformed to its persistent counterpart before storage.

With OpenID 1.1, Relying Parties are expected to use the canonicalised form of what the user enters to identify them. It is possible to redirect the human friendly identifier to a persistent one, but that is not particularly nice if you are trying to co-locate the user’s home page and OpenID.

OpenID 2.0: XRIs

The only solution to this problem in earlier drafts of OpenID 2.0 was to use XRIs. When resolving an XRI, the resulting XRDS document includes a persistent identifier in the element.

For example, resolving “=foo” gives us a canonical ID of “=!4EFC.841C.8012.E2F8″. If a user logs in to an RP with the former, the RP will record the latter. This means the following:

  1. If the user stops paying their $12/year and someone else registers “=foo”, that new user will have a different persistent ID so won’t be able to assume the identity.
  2. If the user registers another XRI pointing at the same persistent identifier, it will be considered equivalent.

OpenID 2.0: URL identifiers

But if you want to use URLs as identifiers, how do you solve the problem?

One solution that was shot down was to allow the <CanonicalID> element in the XRDS document for a URL OpenID. Apparently this was rejected because it would result in another round trip during the discovery process to find the endpoint for the persistent ID.

Instead, a feature was added to help detect the case where an identifier was recycled. As part of the positive authentication response, an OP is allowed to modify the claimed ID to include a fragment URI component. If the identifier gets reassigned, the OP is expected to return a different fragment.

This solves problem (1) but not problem (2). As it stands, the OpenID 2.0 specification doesn’t provide much guidance in letting a user change their human friendly URL identifier while maintaining the same identity.

A Solution

One solution to this problem is to make use of the directed identity feature of OpenID 2.0. Rather than making the user’s homepage their identifier, make it an OP identifier URL. This lets the OP decide on the final claimed identifier.

This allows the user to enter their home page (e.g. http://example.com/james), and have the RP record a persistent identifier (e.g. http://example.com/id/42). If the user changes their human friendly identifier, they’ll still be able to use existing services.

This solution does have a few downsides though:

  • Users can log in with any other user’s homepage URL since they all point at the same OP.
  • Supporting both OpenID 1.1 and 2.0 on the same URL will likely cause confusion, since 1.1 requests would record the human friendly identifier and 2.0 requests record the persistent identifier. If an RP upgraded to the 2.0 protocol, the user would appear to be a different person (which is one of the problems we are trying to avoid).

So it seems that there isn’t a good solution if you need to support OpenID 1.1. If anyone else has ideas, I’d be glad to hear them.

By James Henstridge | Posted in Uncategorised | Tagged | Comments (8)

Beer Pouring Machine

27 October, 2007 – 1:58 pm

One of the novelties in the airport lounge at Narita was a beer pouring machine. It manages to consistently pour a good glass of beer every time. You start by placing the glass in the machine:

Beer machine (1)

When you press the start button, it tilts the glass and pours the beer down the side of the glass:

Beer machine (2)

After filling the glass the machine tilts the glass upright again and some extra foam comes out of the second nozzle:

Beer machine (3)

Not only was the machine fun to watch, but the beer was okay too.

By James Henstridge | Posted in Uncategorised | Tagged , , | Comments (4)

On the way to Boston

27 October, 2007 – 11:43 am

I am at Narita Airport at the moment, on the way to Boston for some of the meetings being held during UDS. It’ll be good to catch up with everyone again.

Hopefully this trip won’t be as eventful as the previous one to Florida :)

By James Henstridge | Posted in Uncategorised | Tagged , | Comments Off

OpenID 2.0

23 October, 2007 – 2:24 pm

Most people have probably seen or used OpenID. If you have used it, then it has most likely that it was with the 1.x protocol. Now that OpenID 2.0 is close to release (apparently they really mean it this time …), it is worth looking at the new features it enables. A few that have stood out to me include:

  • proper extension support
  • support for larger requests/responses
  • directed identity
  • attribute exchange extension
  • support for a new naming monopoly

I’ll now discuss each of these in a bit more detail

Extension Support

OpenID 1.1 had one well known extension: the Simple Registration Extension. An OpenID relying party (RP) would send a request with an openid.sreg.required field, and get back user information in openid.sreg.* fields from the OpenID Provider (OP). The RP and OP would just need to know that “openid.sreg” fields means that the simple registration extension is being used.

But what if I want to define my own extension? If my RP sends openid.foo.* fields, how does the OP know that it refers to my extension and not some other extension that happened to pick the same prefix?

OpenID 2.0 solves this problem by borrowing the idea of name space URIs from XML. If I am sending some openid.foo.* fields in an OpenID message, then I also need to send an openid.ns.foo field set to a URI that identifies the extension. This means that a message that sends the same data as openid.bar.* fields should be treated the same provided that openid.ns.bar is set to the extension’s name space URI.

As with XML name spaces, this allows us to piggy back on top of DNS as a way of avoiding conflicts.

Large Requests and Responses

OpenID 1.1 uses HTTP redirects as a way of transferring control between the RP and OP (and vice versa). This means that the upper limit on a message is effectively the same as the smallest upper limit on length of URLs in common web browsers and servers. Internet Explorer seems to have the lowest limit—2,083 characters—so it sets the effective limit on message size.

For simple authentication checks (what OpenID was originally designed for), this is not generally a problem. But once you start to introduce a few extensions, this limit can easily be reached.

OpenID 2.0 allows messages to be sent as an HTTP POST body which effectively removes the upper limit. The recommended way of achieving this is by sending a page to the user’s browser that contains a form that posts to the appropriate endpoint and contains the data as hidden form fields. The form would then get submitted by a JavaScript onload handler.

Directed Identity

For OpenID 1.1, the authentication process goes something like this:

  1. the user enters their identity URL into a form on the RP
  2. the RP performs discovery on that URL to find the user’s OP.
  3. the RP initiates an OpenID authentication request with that OP.

With OpenID 2.0, the discovery process may tell the RP that the URL identifies the OP rather than the user. If this happens, the RP proceeds with the authentication request using the special “http://specs.openid.net/auth/2.0/identifier_select” value as the identity URL. The OP will then fill in the user’s actual identity URL in the subsequent authentication response. As an additional step, the RP is then required to perform discovery on this URL to ensure that the OP is entitled to authenticate it.

There are a number of cases where this feature can be useful:

  1. An OpenID provider can give their users a single URL that will work for everyone. For instance, if AOL sets things up correctly, you’d be able to type “aol.com” into any OpenID 2.0 enabled site to log in with an AIM screen name.
  2. A privacy concious user could configure their own OpenID provider that will hand out different identity URLs to different RPs, similar to how some people use single-purpose email addresses today.
  3. If an RP requires that users use a particular OP, they could use directed identity to begin the authentication request without requiring the user to enter an identity URL.

Attribute Exchange Extension

The OpenID Attribute Exchange extension is like the simple registration extension on steroids. The major differences are:

  • Unlike the simple registration extension, the attribute exchange extension does not have a fixed set of attributes that can be transmitted. Instead it uses URIs to identify the attribute types, making it easy to define new attributes without causing conflicts. Of course an attribute is not particularly useful if no one else supports it, so there is a process set up to standardise common attribute types.
  • As well as receiving attribute values as part of an authentication response, an RP can request that an OP store certain attribute values. This is done as part of an authentication request, so the OP can verify that the user really wants to store the values.
  • The RP can request ongoing updates for the attributes it is interested in. As an example, if you stored your hackergotchi with your OP, changes to the image could be automatically pushed out to all sites you use that want to display that image.

Prop Up A New Naming Monopoly

With OpenID 2.0, a user is supposed to be able to enter an i-name in place of an identity URL in an RP, and be authenticated against the i-broker managing that name. So rather than entering an ugly URL, users can enter an ugly string starting with “=” or “@”.

All it costs to take advantage of this is US$12 per year (or US$55 for an organisation name). They claim that it will be possible to use an i-name in many contexts in the future, but for now it appears to be limited to (1) a subset of OpenID RPs, (2) a web form that people can use to send you emails and (3) an HTTP redirection to your website.

At this point, it seems that i-name support in OpenID is more important to the i-name crowd than the OpenID crowd. That said, the complexity is hidden by most of the existing OpenID libraries, so it’ll most likely get implemented by default on most RPs moving forward.

Conclusion

Overall OpenID 2.0 looks like a worthwhile upgrade, even if some parts like i-names are questionable.

Assuming the attribute exchange extension takes off, it should provide a much richer user experience. Imagine being able to update your shipping address in one place when you move house and having all the online retailers you use receive the updated address immediately. Or changing your email address and having all the bugzilla instances you use pick up the new address instantly (perhaps requiring you to verify the new address first, of course).

The improved extension support should also make it easier for people to experiment with new extensions without accidentally conflicting with each other, which should accelerate development of new features.

By James Henstridge | Posted in Uncategorised | Tagged | Comments (9)

Back from Dunedin

18 October, 2007 – 12:14 pm

Last week I was in sunny Dunedin for a Launchpad/Bazaar integration sprint with Tim and Jonathan. Some of the smaller issues we addressed should make their way to users in the next Launchpad release (these were mainly fixes to confusing error messages on bazaar.launchpad.net). Some of the others will probably only become available a release or two further on (mostly related to improving development workflow for branches hosted on Launchpad).

My previous trip to New Zealand had also been to Dunedin (for last year’s linux.conf.au). Since then they’d replaced all the coins for denominations less than NZ$1. Other than being less familiar to Australians, the smaller coins seem like a good idea. They don’t seem to have taken Australia’s lead in making the $2 coin smaller than the $1 coin though.

By James Henstridge | Posted in Uncategorised | Comments (2)

Google’s Australian Election Tools

4 October, 2007 – 8:08 pm

It is probably old news to some, but Google have put up an information page on the upcoming Australian Federal Election.

The most useful tool is the Google Maps overlay that provides information about the different electorates. At the moment it only has information about the sitting members, their margin and links to relevant news articles. Presumably more information will become available once the election actually gets called.

Presumably they are planning on offering similar tools for next year’s US elections and this is a beta. So even if you aren’t interested in Australian politics, it might be worth a peak to see what is provided.

By James Henstridge | Posted in Uncategorised | Comments Off

Signed Revisions with Bazaar

4 October, 2007 – 12:26 am

One useful feature of Bazaar is the ability to cryptographically sign revisions. I was discussing this with Ryan on IRC, and thought I’d write up some of the details as they might be useful to others.

Anyone who remembers the past security of GNOME and Debian servers should be able to understand the benefits of being able to verify the integrity of a source code repository after such an incident. Rather than requiring all revisions made since the last known safe backup to be examined, much of the verification could be done mechanically.

Turning on Revision Signing

The first thing you’ll need to do is get a PGP key and configure GnuPG to use it. The GnuPG handbook is a good reference on doing this. As the aim is to provide some assurance that the revisions you publish were really made by you, it’d be good to get the key signed by someone.

Once that is done, it is necessary to configure Bazaar to sign new revisions. The easiest way to do this is to edit ~/.bazaar/bazaar.conf to look something like this:

[DEFAULT]
email = My Name <me@example.com>
create_signatures = always

Now when you run “bzr commit“, a signature for the new revision will be stored in the repository. With this configuration change, you will be prompted for your pass phrase when making commits. If you’d prefer not to enter it repeatedly, there are a few options available:

  1. install gpg-agent, and use it to remember your pass phrase in the same way you use ssh-agent.
  2. install the gnome-gpg wrapper, which lets you remember your pass phrase in your Gnome keyring. To use gnome-gpg, you will need to add an additional configuration value: “gpg_signing_command = gnome-gpg“.

Signatures are transferred along with revisions when you push or pull a branch, perform merges, etc.

How Does It Work?

So what does the signature look like, and what does it cover? There is no command for printing out the signatures, but we can access them using bzrlib. As an example, lets look at the signature on the head revision of one of my branches:

>>> from bzrlib.branch import Branch
>>> b = Branch.open('http://bazaar.launchpad.net/~jamesh/storm/reconnect')
>>> b.last_revision()
'james.henstridge@canonical.com-20070920110018-8e88x25tfr8fx3f0'
>>> print b.repository.get_signature_text(b.last_revision())
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

bazaar-ng testament short form 1
revision-id: james.henstridge@canonical.com-20070920110018-8e88x25tfr8fx3f0
sha1: 467b78c3f8bfe76b222e06c71a8f07fc376e0d7b
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFG8lMHAa+T2ZHPo00RAsqjAJ91urHiIcu4Bim7y1tc5WtR+NjvlACgtmdM
9IC0rtNqZQcZ+GRJOYdnYpA=
=IONs
-----END PGP SIGNATURE-----

>>>

If we save this signature to a file, we can verify it with a command like “gpg --verify signature.txt” to prove that it was made using my PGP key. Looking at the signed text, we see three lines:

  1. An identifier for the checksum algorithm. This is included to future proof old signatures should the need arise to alter the checksum algorithm at a later date.
  2. The revision ID that the signature applies to. Note that this is the full globally unique identifier rather than the shorter numeric identifiers that are only unique in the context of an individual branch.
  3. The checksum, in SHA1 form.

For the current signing algorithm, the checksum is made over the long form testament for the revision, which can easily be verified:

$ bzr branch http://bazaar.launchpad.net/~jamesh/storm/reconnect
$ cd reconnect
$ bzr testament --long > testament.txt
$ sha1sum testament.txt
467b78c3f8bfe76b222e06c71a8f07fc376e0d7b  testament.txt

Looking at the long form testament, we can see what the signature ultimately covers:

  1. The revision ID
  2. The name of the committer
  3. The date of the commit
  4. The parent revision IDs
  5. The commit message
  6. A list of the files that comprise the source tree for the revision, along with SHA1 sums of their contents
  7. Any revision properties

So if the revision testament matches the revision signature and the revision signature validates, you can be sure that you are looking at the same code as the person who made the signature.

It is worth noting that while the signature makes an assertion about the state of the tree at that revision — the only thing it tells you about the ancestry is the revision IDs of the parents. If you need assurances about those revisions, you will need to check their signatures separately. One of the reasons for this is that you might not know the full history of a branch if it has ghost revisions (as might happen when importing code from certain foreign version control systems).

Signing Past Revisions

If you’ve already been using Bazaar but had not enabled revision signing, it is likely that you’ve got a bunch of unsigned revisions lying around. If that is the case, you can sign the revisions in bulk using the “bzr sign-my-commits” command. It will go through all revisions in the ancestry, and generate signatures for all the commits that match your committer ID.

Verifying Signatures in Bulk

To verify all signatures found in a repository, John Arbash Meinel’s signing plugin can be used, which provides a “bzr verify-sigs” command. It can be installed with the following commands:

$ mkdir -p ~/.bazaar/plugins
$ bzr branch http://bzr.arbash-meinel.com/plugins/signing/ ~/.bazaar/plugins/signing

When the command is run it will verify the integrity of all the signatures, and give a summary of how many revisions each person has signed.

By James Henstridge | Posted in Uncategorised | Tagged | Comments (1)

Schema Generation in ORMs

28 September, 2007 – 6:05 pm

When Storm was released, one of the comments made was that it did not include the ability to generate a database schema from the Python classes used to represent the tables while this feature is available in a number of competing ORMs. The simple reason for this is that we haven’t used schema generation in any of our ORM-using projects.

Furthermore I’d argue that schema generation is not really appropriate for long lived projects where the data stored in the database is important. Imagine developing an application along these lines:

  1. Write the initial version of the application.
  2. Generate a schema from the code.
  3. Deploy one or more instances of the application in production, and accumulate some data.
  4. Do further development on the application, that involves modifications to the schema.
  5. Deploy the new version of the application.

In order to perform step 5, it will be necessary to modify the existing database to match the new schema. These changes might be in a number of forms, including:

  • adding or removing a table
  • adding or removing a column from a table
  • changing the way data is represented in a particular column
  • refactoring one table into two related tables or vice versa
  • adding or removing an index

Assuming that you want to keep the existing data, it isn’t enough to simply represent the new schema in the updated application: we need to know how that new schema relates to the old one in order to migrate the existing data.

For some changes like addition of tables, it is pretty easy to update the schema given knowledge of the new schema. For others it is more difficult, and will often require custom migration logic. So it is likely that you will need to write a custom script to migrate the schema and data.

Now we have two methods of building the database schema for the application:

  1. generate a schema from the new version of the application.
  2. generate a schema from the old version of the application, then run the migration script.

Are you sure that the two methods will result in the same schema? How about if we iterate the process another 10 times or so? As a related question, are you sure that the database environment your tests are running under match the production environment?

The approach we settled on with Launchpad development was to only deal with migration scripts and not generate schemas from the code. The migration scripts are formulated as a sequence of SQL commands to migrate the schema and data as needed. So to set up a new instance, a base schema is loaded then patched up to the current schema. Each patch leaves a record in the database that it has been applied so it is trivial to bring a database up to date, or check that an application is in sync with the database.

When the schema is not generated from the code, it also means that the code can be simpler. As far as Python ORM layer is concerned, does it matter what type of integer a field contains? Does the Python code care what indexes or constraints are defined for the table? By only specifying what is needed to effectively map data to Python objects, we end up with easy to understand code without annotations that probably can’t specify everything we want anyway.

By James Henstridge | Posted in Uncategorised | Tagged , , | Comments (6)