Docker – scrambled tofu

Fixing botched migrations with `oc debug`

When using Openshift Origin to deploy software, you often have your containers execute a database migration as part of their deployment, e.g. in your Dockerfile:

CMD ./manage.py migrate --noinput && \
    gunicorn -w 4 -b 0.0.0.0:8000 myapp.wsgi:application

This works great until your migration won’t apply cleanly without intervention, your newly deploying pods are in crashloop backup, and you need to understand why. This is where the `oc debug` command comes in. Using `oc debug` we can ask for a shell on a pod running or newly created.

Assuming we have a deployment config `frontend`:

oc debug dc/frontend

will give us a shell in a running pod for the latest stable deployment (i.e. your currently running instances, not the ones that are crashing).

However let’s say deployment #44 is the one crashing. We can debug a pod from the replication controller for deployment #44.

oc debug rc/frontend-44

will give us a shell in a new pod for that deployment, with our new code, and allows us to manually massage our broken migration in (e.g. by faking the data migration that was retroactively added for production).

Running Django on Docker: a workflow and code

It has been an extremely long time between beers (10 months!). I’ve gotten out of the habit of blogging and somehow I never blogged about the talk I co-presented at PyCon AU this year on Pallet and Forklift the standard and tool we’ve developed at Infoxchange to help make it easier to develop web-applications on Docker ((There’s also Straddle Carrier, a set of Puppet manifests for loading Docker containers on real infrastructure, but they’ve not been released yet as they rely too much on our custom Puppet config.)).

Infoxchange is one of the few places I’m aware of that runs Docker in prod. If you’re looking at using Docker to do web development, it’s worth checking out what we’ve been doing over on the Infoxchange devops blog.

A libnss plugin for Docker

So we’ve been starting to use Docker at Infoxchange. A little while ago my Infoxchange colleague Ricky Cook wrote a Twisted service to resolve Docker container names into their private IP addresses.

I thought this was a pretty neat idea, although you had to do some things like return SERVFAIL to round-robin to another DNS server if it wasn’t a Docker host. However it got problematic when something like dhclient/NetworkManager was in charge of your resolv.conf.

I realised on Friday night the the Truely UNIX solution was to write an NSS plugin. This was not quite as trivial as I thought it would be. However I spent a bit of time staring at Lennart’s mdns plugin and started writing one of my own.

docker-nss should work well enough to resolve container IDs and temporary assigned names that you get from docker ps.

There’s a bit of work to do. It can’t reverse IPs back into container names yet. Nor does it return a full list of aliases for a container. Nor does it return a list of IPs for an image name. Patches welcome! It currently depends on GLib, which drags GLib into the memory space of every process on your system, however it wouldn’t be too hard to port away from it (I’m just lazy).

It has unit tests via a mocked docker client. You do have to have modified your nsswitch.conf in order to be able to run the tests. Although if this really bothers someone we could make the test suite itself a Docker container.