Running Django on Docker: a workflow and code

It has been an extremely long time between beers (10 months!). I’ve gotten out of the habit of blogging and somehow I never blogged about the talk I co-presented at PyCon AU this year on Pallet and Forklift the standard and tool we’ve developed at Infoxchange to help make it easier to develop web-applications on Docker1.

Infoxchange is one of the few places I’m aware of that runs Docker in prod. If you’re looking at using Docker to do web development, it’s worth checking out what we’ve been doing over on the Infoxchange devops blog.

  1. There’s also Straddle Carrier, a set of Puppet manifests for loading Docker containers on real infrastructure, but they’ve not been released yet as they rely too much on our custom Puppet config. []
Posted in Uncategorized | Leave a comment

lazy-loading class-based-views in Django

So one of the nice things with method-based views in Django is the ability to do this sort of thing to load a view at the path frontend.views.home:

urlpatterns = patterns(
    'frontend.views',

    url(r'^$', 'home', name='home'),
)

Unfortunately, if you’re using class-based-views, you can’t do this:

urlpatterns = patterns(
    'frontend.views',

    url(r'^$', 'HomeView', name='home'),
)

And instead you had to resort to importing the view and calling HomeView.as_view(). Sort of annoying when you didn’t want to import all of those views.

It turns out however that overloading the code to resolve HomeView is not that difficult, and we can do it with a pretty straightforward monkeypatch. This version uses the kwargs argument of url() to pass keyword arguments to as_view().

from django.conf import urls
from django.views.generic import View


class ClassBasedViewURLPattern(urls.RegexURLPattern):
    """
    A version of RegexURLPattern able to handle class-based-views

    Monkey-patch it in to support class-based-views
    """

    @property
    def callback(self):
        """
        Hook locating the view to handle class based views
        """

        view = super(ClassBasedViewURLPattern, self).callback

        if isinstance(view, type) and issubclass(view, View):
            view = view.as_view(**self.default_args)
            self.default_args = {}

        return view

urls.RegexURLPattern = ClassBasedViewURLPattern
Posted in Uncategorized | Comments Off

A libnss plugin for Docker

So we’ve been starting to use Docker at Infoxchange. A little while ago my Infoxchange colleague Ricky Cook wrote a Twisted service to resolve Docker container names into their private IP addresses.

I thought this was a pretty neat idea, although you had to do some things like return SERVFAIL to round-robin to another DNS server if it wasn’t a Docker host. However it got problematic when something like dhclient/NetworkManager was in charge of your resolv.conf.

I realised on Friday night the the Truely UNIX solution was to write an NSS plugin. This was not quite as trivial as I thought it would be. However I spent a bit of time staring at Lennart’s mdns plugin and started writing one of my own.

docker-nss should work well enough to resolve container IDs and temporary assigned names that you get from docker ps.

There’s a bit of work to do. It can’t reverse IPs back into container names yet. Nor does it return a full list of aliases for a container. Nor does it return a list of IPs for an image name. Patches welcome! It currently depends on GLib, which drags GLib into the memory space of every process on your system, however it wouldn’t be too hard to port away from it (I’m just lazy).

It has unit tests via a mocked docker client. You do have to have modified your nsswitch.conf in order to be able to run the tests. Although if this really bothers someone we could make the test suite itself a Docker container.

Posted in Uncategorized | Comments Off

Django utility methods (including New Relic deployment notification)

So we’ve moved to Github here at Infoxchange as our primary development platform because pull requests and Travis CI are much nicer than yelling across the room at each other1. To enable Travis to build our code, we’ve needed to move our little utility libraries to Github too. Since some of these were already on pip, it made sense to open source the rest of them too.

The most useful is a package called IXDjango which includes a number of generally useful management commands for Django developers. Especially useful are deploy which will run a sequence of other commands for deployment, and newrelic_notify_deploy which you can use to notify New Relic of your deployment, which annotates all of your graphs with the version number.

We hope these are useful to people.

  1. big shout out to both Github and Travis CI for supporting our not-for-profit mission with gratis private accounts []
Posted in django | Comments Off

Writing your first web app using Python and Flask

I presented a tutorial at linux.conf.au a couple of weeks ago on what there was for Python developers between CGI scripts and Django. That developers needn’t still be writing CGI scripts in 2014 (it happens) and there were frameworks that met your needs.

This tutorial introduces the microframework Flask, and shows of a whole bunch of things you can do with it, up to being a fully-fledged replacement for Django if you’re so inclined.

Video
Examples source

In other news, I am now a maintainer of Lettuce, a BDD framework for Python/Django. So expect a few more Lettuce related blog posts (if I stop ignoring my blog).

Posted in Uncategorized | 2 Comments

More than a side salad: behaviour driven testing and test driven design in Django with Lettuce

Been quiet lately because I am super busy getting a project out of the door. However I did find time to give this talk last night on behaviour driven testing with Lettuce at MelbDjango.

Apologies for the glitches in the PDF. reveal.js is amazing for doing presentations but I didn’t have time to fix the glitches in the PDF output. The presentation is also able to be cloned from Git (view index.html), but you require the fonts (they’re all libre but I was too lazy to use webfonts).

Hopefully, when I get a breather, I can write about some other code I’ve written, or am writing. Or maybe catch up on the rest of my life/projects.

Posted in best practice, django, python | 3 Comments

Testing sites with beforeunload and Lettuce/Cucumber

I recently added a beforeunload event handler to a site I’m working on, which instantly caused a regression of the entire Lettuce test suite before they got stuck on a “Leave this page?” dialog. We reuse the same Selenium browser session between tests in order to make our tests run in something approximating less than a decade.

Unfortunately Ghostdriver can’t see alerts and dialogs, which makes simply dismissing the dialog in Selenium kind of hard, but an easier way is at the end of the scenario to simply disable the event, and let it be reinstated with the next page load.

Add this hook to your Lettuce steps:

@after.each_scenario
def disable_beforeunload(scenario):
    """
    Disable before unload after a scenario so that the next scenario can
    reload the site.
    """

    world.browser.execute_script("""
try {
    $(window).off('beforeunload');
} catch (e) {
}
    """)
Posted in Uncategorized | Comments Off

vim + pathogen + git submodules

As part of PyCon Au this weekend I did a lot of hacking on my laptop, which is not something I’ve done for a while, given how much I used to hack on my laptop. I was frequently getting annoyed at how my vim config wasn’t the same as it’s currently on my work desktop.

Back in uni, I used to keep my dotfiles in revision control on a machine I could connect to. All I needed was my ssh agent and I could get all my config.

Recently, when wanting to extend my vim config, people’s pages tell me to do it through vim pathogen, which I didn’t have set up.

Also there have been times when people have asked for my vim setup, which I wasn’t easily able to provide.

Given all of that, I decided it was time to rebuild my config using pathogen + git submodules from the ground up. As part of this, I updated quite a few plugins, and started using a few new things I hadn’t had available without pathogen, so it’s still a bit of a work in progress, but it’s here.

Installing new plugins is easy with git submodule add PATH bundle/MODULE_NAME.

If you’re a vim user, I strongly recommend this approach.

Posted in Uncategorized | 3 Comments

free as in gorgeous

I’m at PyCon Au.1 I made it this year. There were no unexpected collisions in the week leading up to it.

I decided at the last moment to do a lightning talk on a piece of Django-tech I put together at Infoxchange, a thing called Crisper, which I use in a very form-heavy app I’m developing. Crisper is going to be up on Github, just as soon as I have time to extract it from the codebase it’s part of (which mostly consists of working out where it’s pointlessly coupled).

The title of my post relates to deciding to do this talk, and the amazing free and open graphic design assets I have available at my fingertips to create an instant title slide. 5 minutes in Inkscape, using the Tango palette and Junction (available in Fedora as tlomt-junction-fonts) and Inkscape to put together things like this:

crisper

Anyway so some pretty good content today at DjangoCon Au. I especially enjoyed the food for thought about whether we should be targeting new development at Python 3. It has struck me that writing greenfields code on Python 2 is a dead end that will eventually result in my having to port it anyway. Thinking I will conduct a quick stocktake of the dependencies we’re using at work to evaluate a Python 3 port. I think generally we’re writing pretty compatible code using modern Python syntax, so from the point of view of our apps, at port at this stage would be fairly straightforward.

Some interesting discussion too on the future of frameworks like Django. Is the future microframeworks + rich JS on the frontend using Angular or Ember. How do things like Django adapt to that? Even though JS has nothing on Python, is the future node.js on the back and somethingJS on the front?2

~

This is actually a nice aside that I’ve been working on a pure in-web app for visualising data from Redmine using D3.js and Redmine’s RESTful API.3 Intending to add configuration in localstorage (probably via Angular) once I have the visualisations looking good. It both serves a practical purpose and an interesting experiment on writing web apps without a web server.

~

Almost back on topic, I had dinner with a bunch of cyclists tonight, and got to talking about my crowd-sourced cycle maps project,4 which hasn’t gone anywhere because I still don’t have any hosting for it. I should try having dinner with cyclists who work for PaaS hosting providers.5

Finally, I’m trying to recruit talented web developers who have a love of Agile, test driven design and Python. Also a technical tester. Come and talk to me.

  1. Thanks to Infoxchange, who sent me, and are also sponsoring the conference. []
  2. It always cracked me up that my previous web-app was this HTML5-y app, but designed without the use of frameworks. Old and new both at once. []
  3. https://github.com/danni/redmine-viz []
  4. https://github.com/danni/cycle-router []
  5. I need PostGres 9.2 and PostGIS 2.0. Come chat to me! []
Posted in Uncategorized | Comments Off

calling Melbourne Python and Perl programmers

InfoxchangeFor the last couple of months I’ve been working for Infoxchange, a not-for-profit that provides technology to other not for profit. I’ve been working in the webapp development team, where we mostly work on webapps in the health and community sector using both Perl (for the older stuff) and Python/Django (for the newer stuff).

The government didn’t really work out for me, so this is a nice change. It’s relaxed, people wear jeans to work. We have fair trade tea and coffee and a delivery of CSA fruit every week.

It is busy though. We’ve got a lot going on, and not enough people to do it, so we’re looking for more. So if you’re a committed, talented Perl, Python or Web/Javascript programmer or devops who is in (or willing to move to) Melbourne, who wants to make a difference, you should get in touch with me. The pay is good (you will not be working for peanuts) and the team is fun.

We love open source, we contribute upstream, we have an organisation Github account. You can run what you want on your desktop. Developers have technical ownership over their work. Development is Agile.

Some technologies we love are Perl, Mojolicious, Python, Django, Javascript, jQuery, Bootstrap, Less, NodeJS, AngularJS, Github, Puppet, Debian and PostgreSQL. If you love any of those too, you should totally get in touch. If you love more technologies, bring those along too.

Posted in Uncategorized | Comments Off