Schema Generation in ORMs

When Storm was released, one of the comments made was that it did not include the ability to generate a database schema from the Python classes used to represent the tables while this feature is available in a number of competing ORMs. The simple reason for this is that we haven’t used schema generation in any of our ORM-using projects.

Furthermore I’d argue that schema generation is not really appropriate for long lived projects where the data stored in the database is important. Imagine developing an application along these lines:

  1. Write the initial version of the application.
  2. Generate a schema from the code.
  3. Deploy one or more instances of the application in production, and accumulate some data.
  4. Do further development on the application, that involves modifications to the schema.
  5. Deploy the new version of the application.

In order to perform step 5, it will be necessary to modify the existing database to match the new schema. These changes might be in a number of forms, including:

  • adding or removing a table
  • adding or removing a column from a table
  • changing the way data is represented in a particular column
  • refactoring one table into two related tables or vice versa
  • adding or removing an index

Assuming that you want to keep the existing data, it isn’t enough to simply represent the new schema in the updated application: we need to know how that new schema relates to the old one in order to migrate the existing data.

For some changes like addition of tables, it is pretty easy to update the schema given knowledge of the new schema. For others it is more difficult, and will often require custom migration logic. So it is likely that you will need to write a custom script to migrate the schema and data.

Now we have two methods of building the database schema for the application:

  1. generate a schema from the new version of the application.
  2. generate a schema from the old version of the application, then run the migration script.

Are you sure that the two methods will result in the same schema? How about if we iterate the process another 10 times or so? As a related question, are you sure that the database environment your tests are running under match the production environment?

The approach we settled on with Launchpad development was to only deal with migration scripts and not generate schemas from the code. The migration scripts are formulated as a sequence of SQL commands to migrate the schema and data as needed. So to set up a new instance, a base schema is loaded then patched up to the current schema. Each patch leaves a record in the database that it has been applied so it is trivial to bring a database up to date, or check that an application is in sync with the database.

When the schema is not generated from the code, it also means that the code can be simpler. As far as Python ORM layer is concerned, does it matter what type of integer a field contains? Does the Python code care what indexes or constraints are defined for the table? By only specifying what is needed to effectively map data to Python objects, we end up with easy to understand code without annotations that probably can’t specify everything we want anyway.

Upgrading to Ubuntu Gutsy

I got round to upgrading my desktop system to Gutsy today. I’d upgraded my laptop the previous week, so was not expecting much in the way of problems.

I’d done the original install on my desktop back in the Warty days, and the root partition was a bit too small to perform the upgrade. As there was a fair bit of accumulated crud, I decided to do a clean install. Things mostly worked, but there were a few problems, which I detail below:

Dual Head Configuration

With previous releases, I was using the Radeon driver’s MergedFB mode, as it gives a better user experience than the traditional Xinerama code (3D acceleration on both heads, better performance, etc). After moving adding the MergedFB options to xorg.conf, I was just getting the same image cloned on both displays.

Looking at the X server log file, there was a message saying that MergedFB support had been removed in favour of RandR 1.2 support. And it was possible to get dual head working with the xrandr command line tool:

xrandr --output VGA-0 --right-of DVI-0

It was good to know that dual-head still worked, but I didn’t want to reconfigure this every time I restarted the machine. I didn’t find much information on how to configure the initial RandR configuration on the X.org website, but did find a useful guide on the Intel Linux Graphics website. While the guide was aimed at the Intel driver, it had enough information to get things configured for the Radeon driver. The main difference was in the naming of the outputs. Below is a an excerpt of my configuration file that configures things the way I had them previously:

Section "Device"
        Identifier      "ATI Technologies Inc RV280 [Radeon 9200 SE]"
        Driver          "ati"
        BusID           "PCI:1:0:0"
        Option          "monitor-DVI-0" "Sony SDM-S74 [1]"
        Option          "monitor-VGA-0" "Sony SDM-S74 [2]"
EndSection

Section "Monitor"
        Identifier      "Sony SDM-S74 [1]"
        Option          "DPMS"
        HorizSync       30-65
        VertRefresh     50-75
        Option          "LeftOf"        "Sony SDM-S74 [2]"
EndSection

Section "Monitor"
        Identifier      "Sony SDM-S74 [2]"
        Option          "DPMS"
        HorizSync       30-65
        VertRefresh     50-75
EndSection

Section "Screen"
        Identifier      "Default Screen"
        Device          "ATI Technologies Inc RV280 [Radeon 9200 SE]"
        Monitor         "Sony SDM-S74 [1]"
        DefaultDepth    16
        SubSection "Display"
                Modes           "1280x1024" "1024x768" "800x600" "640x480"
                Virtual         2560 1024
        EndSubSection
EndSection

I had originally tried setting the VGA monitor to be “RightOf” the monitor connected to the DVI, but that left me with the desktop in clone mode. The main difference I’ve noticed with this configuration compared to my previous one is that the GDM login prompt displays on the right hand head (VGA) rather than the left hand head (DVI).

Window Shadows Don’t Render

Desktop Effects were enabled by default after the install (and on the live CD). While some effects seemed to work, the shadows on the panel and drop down menus were rendered as opaque grey boxes around the windows. I ended up just disabling the effects to clear up the problem.

This bug had already been reported as bug 141304 (which may be the same as bug 116808).

Firefox Crashes on Startup

When I tried to start firefox, it would momentarily display a window and then crash. This appears to be bug 133124, and seems to only occur on AMD64 systems. The problem appears to be in the ubuntulooks theme engine, and switching to a different control theme makes the problem go away, but hopefully it’ll get fixed for the final release.

Problems Rendering Ligatures in Firefox

The problems rendering ligatures in firefox seem to be back again. This problem was never really fixed, but was worked around by removing the ligature table entries from the DejaVu fonts. With the ligature table entries back, the symptoms have returned. This is bug 37828.

Manic Times

When I got back from Florida, I found a copy of the Manic Times in the mail. It seems that I received the copy because I used to be subscribed to The Chaser back when it was a newspaper. The newspaper is being edited by Charles Firth, who was the US correspondent in the last series of The Chaser’s War on Everything.

The content is fairly different to what was published in The Chaser, in that they are nominally true. That said, they are written from a non-mainstream point of view. The issue I received seemed to focus on the recent APEC conference and related security measures (which appear to have been fairly poor).

I haven’t yet decided whether to subscribe, but the online subscription form thoughtfully includes “Governor-General”, “Deputy Vice-President”, “The Right Hon”, “Queen” and “Archbishop” in the dropdown list of titles to pick from – something that most likely inconveniences people on other web sites.