scrambled tofu – Page 3 – Danielle's technical stuffs

Django utility methods (including New Relic deployment notification)

So we’ve moved to Github here at Infoxchange as our primary development platform because pull requests and Travis CI are much nicer than yelling across the room at each other ((big shout out to both Github and Travis CI for supporting our not-for-profit mission with gratis private accounts)). To enable Travis to build our code, we’ve needed to move our little utility libraries to Github too. Since some of these were already on pip, it made sense to open source the rest of them too.

The most useful is a package called IXDjango which includes a number of generally useful management commands for Django developers. Especially useful are deploy which will run a sequence of other commands for deployment, and newrelic_notify_deploy which you can use to notify New Relic of your deployment, which annotates all of your graphs with the version number.

We hope these are useful to people.

Writing your first web app using Python and Flask

I presented a tutorial at linux.conf.au a couple of weeks ago on what there was for Python developers between CGI scripts and Django. That developers needn’t still be writing CGI scripts in 2014 (it happens) and there were frameworks that met your needs.

This tutorial introduces the microframework Flask, and shows of a whole bunch of things you can do with it, up to being a fully-fledged replacement for Django if you’re so inclined.

Writing your first web app using Python and Flask from Danielle Madeley

Video
Examples source

In other news, I am now a maintainer of Lettuce, a BDD framework for Python/Django. So expect a few more Lettuce related blog posts (if I stop ignoring my blog).

More than a side salad: behaviour driven testing and test driven design in Django with Lettuce

Been quiet lately because I am super busy getting a project out of the door. However I did find time to give this talk last night on behaviour driven testing with Lettuce at MelbDjango.

More than a side salad: behaviour driven testing and test driven design in Django with Lettuce from Danielle Madeley

Apologies for the glitches in the PDF. reveal.js is amazing for doing presentations but I didn’t have time to fix the glitches in the PDF output. The presentation is also able to be cloned from Git (view index.html), but you require the fonts (they’re all libre but I was too lazy to use webfonts).

Hopefully, when I get a breather, I can write about some other code I’ve written, or am writing. Or maybe catch up on the rest of my life/projects.

Testing sites with beforeunload and Lettuce/Cucumber

I recently added a beforeunload event handler to a site I’m working on, which instantly caused a regression of the entire Lettuce test suite before they got stuck on a “Leave this page?” dialog. We reuse the same Selenium browser session between tests in order to make our tests run in something approximating less than a decade.

Unfortunately Ghostdriver can’t see alerts and dialogs, which makes simply dismissing the dialog in Selenium kind of hard, but an easier way is at the end of the scenario to simply disable the event, and let it be reinstated with the next page load.

Add this hook to your Lettuce steps:

@after.each_scenario
def disable_beforeunload(scenario):
    """
    Disable before unload after a scenario so that the next scenario can
    reload the site.
    """

    world.browser.execute_script("""
try {
    $(window).off('beforeunload');
} catch (e) {
}
    """)

vim + pathogen + git submodules

As part of PyCon Au this weekend I did a lot of hacking on my laptop, which is not something I’ve done for a while, given how much I used to hack on my laptop. I was frequently getting annoyed at how my vim config wasn’t the same as it’s currently on my work desktop.

Back in uni, I used to keep my dotfiles in revision control on a machine I could connect to. All I needed was my ssh agent and I could get all my config.

Recently, when wanting to extend my vim config, people’s pages tell me to do it through vim pathogen, which I didn’t have set up.

Also there have been times when people have asked for my vim setup, which I wasn’t easily able to provide.

Given all of that, I decided it was time to rebuild my config using pathogen + git submodules from the ground up. As part of this, I updated quite a few plugins, and started using a few new things I hadn’t had available without pathogen, so it’s still a bit of a work in progress, but it’s here.

Installing new plugins is easy with git submodule add PATH bundle/MODULE_NAME.

If you’re a vim user, I strongly recommend this approach.

free as in gorgeous

I’m at PyCon Au. ((Thanks to Infoxchange, who sent me, and are also sponsoring the conference.)) I made it this year. There were no unexpected collisions in the week leading up to it.

I decided at the last moment to do a lightning talk on a piece of Django-tech I put together at Infoxchange, a thing called Crisper, which I use in a very form-heavy app I’m developing. Crisper is going to be up on Github, just as soon as I have time to extract it from the codebase it’s part of (which mostly consists of working out where it’s pointlessly coupled).

The title of my post relates to deciding to do this talk, and the amazing free and open graphic design assets I have available at my fingertips to create an instant title slide. 5 minutes in Inkscape, using the Tango palette and Junction (available in Fedora as tlomt-junction-fonts) and Inkscape to put together things like this:

Anyway so some pretty good content today at DjangoCon Au. I especially enjoyed the food for thought about whether we should be targeting new development at Python 3. It has struck me that writing greenfields code on Python 2 is a dead end that will eventually result in my having to port it anyway. Thinking I will conduct a quick stocktake of the dependencies we’re using at work to evaluate a Python 3 port. I think generally we’re writing pretty compatible code using modern Python syntax, so from the point of view of our apps, at port at this stage would be fairly straightforward.

Some interesting discussion too on the future of frameworks like Django. Is the future microframeworks + rich JS on the frontend using Angular or Ember. How do things like Django adapt to that? Even though JS has nothing on Python, is the future node.js on the back and somethingJS on the front? ((It always cracked me up that my previous web-app was this HTML5-y app, but designed without the use of frameworks. Old and new both at once.))

This is actually a nice aside that I’ve been working on a pure in-web app for visualising data from Redmine using D3.js and Redmine’s RESTful API. ((https://github.com/danni/redmine-viz)) Intending to add configuration in localstorage (probably via Angular) once I have the visualisations looking good. It both serves a practical purpose and an interesting experiment on writing web apps without a web server.

Almost back on topic, I had dinner with a bunch of cyclists tonight, and got to talking about my crowd-sourced cycle maps project, ((https://github.com/danni/cycle-router)) which hasn’t gone anywhere because I still don’t have any hosting for it. I should try having dinner with cyclists who work for PaaS hosting providers. ((I need PostGres 9.2 and PostGIS 2.0. Come chat to me!))

Finally, I’m trying to recruit talented web developers who have a love of Agile, test driven design and Python. Also a technical tester. Come and talk to me.

First thoughts on RedHat OpenShift

I’m looking for a PaaS provider that isn’t going to cost me very much (or anything at all) and supports Flask and PostGIS. Based on J5’s recommendation in my blog the other day, I created an OpenShift account.

A free account OpenShift gives you three small gears ((small gears have 1GB of disk and 512MB of RAM allocated)) which are individual containers you can run an app on. You can either run an app on a single gear or have it scale to multiple gears with load balancing. You then install components you need, which OpenShift refers to by the pleasingly retro name of cartridges. So for instance, Python 2.7 is one cartridge and PostgreSQL is another. You can either install all cartridges on one gear or on separate gears based on your resource needs ((I think if you have a load balancing (scalable) application, your database needs to be on its own gear so all the other gears can access it.)).

You choose your base platform cartridge (i.e. Python-2.6) and you optionally give it a git URL to do an initial checkout from (which means you can deploy an app that is already arranged for OpenShift very fast). The base cartridge sets up all the hooks for setting up after a git push (you get a git remote that you can push to to redeploy your app). The two things you need are a root setup.py containing your pip requirements, and a wsgi/application file which is a Python blob containing an WSGI object named application. For Python it uses virtualenv and all that awesome stuff. I assume for node.js you’d provide a package.json and it would use npm, similarly RubyGems for Ruby etc.

There’s a nifty command line tool written in Ruby (what happened to Python-only Redhat?) that lets you do all the sort of cloud managementy stuff, including reloading cartridges and gears, tailing app logs and SSHing into the gear. I think an equivalent of dbshell would be really useful based on your DB cartridge, but it’s not a big deal.

There are these deploy hooks you can add to your git repo to do things like create your databases. I haven’t used them yet, but again it would make deploying your app very fast.

There are also quickstart scripts for deploying things like WordPress, Rails and a Jenkins server onto a new gear. Speaking of Jenkins there’s also a Jenkins client cartridge which I think warrants experimentation.

So what’s a bit crap? Why isn’t my app running on OpenShift yet? Basically because the available cartridges are a little antique. The supported Python is Python 2.6, which I could port my app too; or there are community-supported 2.7 and 3.3 cartridges, so that’s fine for me (TBH, I thought my app would run on 2.6) but maybe annoying for others. There is no Celery cartridge, which is what I would have expected, ideally so you can farm tasks out to other gears, and although you apparently can use it, there’s very little documentation I could find on how to get it running.

Really though the big kick in the pants is there is no cartridge for Postgres 9.2/PostGIS 2.0. There is a community cartridge you can use on your own instance of OpenShift Origin, but that defeats the purpose. So either I’m waiting for new Postgres to be made available on OpenShift or backporting my code to Postgres 8.4.

Anyway, I’m going to keep an eye on it, so stay tuned.

Extending geoalchemy through monkeypatching

I’ve been working on the data collection part of my cycle route modelling. I’m hoping that I can, as a first output, put together a map of where people are cycling in Melbourne. A crowd-sourced view of the best places to cycle, if you will. Given I will probably be running this in the cloud ((your recommendations for cloud-based services please, must be able to run Flask and PostGIS and be super cheap)), I thought it was best to actually store the data in a GIS database, rather than lots and lots of flat files.

A quick Google turned up GeoAlchemy, which are GIS extensions for SQLAlchemy. Provides lots of the standard things you want to do as methods on fields, but this is only a limited set of what you can do with PostGIS. Since I’m going to be wanting to do things like binning data, I thought it was worth figuring out how hard it was to call other PostGIS methods.

GeoAlchemy supports subclassing to create new dialects, but you have to subclass 3 classes, and it’s basically a pain in the neck when you just want to extend the functionality of the PostGIS dialect. Probably what I should do is submit a pull request with the rest of the PostGIS API as extensions, but I’m lazy. Henceforth, for the second time this week I am employing monkey patching to get the job done (and for the second time this week, kittens cry).

Functions in GeoAlchemy require two things, a method stub saying how we collect the arguments and the return (look at geoalchemy.postgis.pg_functions) and a mapping from this to the SQL function. Since we only care about one dialect, we can make this easier on ourselves by combining these two things. Firstly we monkeypatch in the method stubs:

from geoalchemy.functions import BaseFunction
from geoalchemy.postgis import pg_functions

@monkeypatchclass(pg_functions)
class more_pg_functions:
    """
    Additional functions to support for PostGIS
    """

    class length_spheroid(BaseFunction):
        _method = 'ST_Length_Spheroid'

Note the _method attribute which isn’t something used anywhere else. We can then patch in support for this:

from geoalchemy.dialect import SpatialDialect

@monkeypatch(SpatialDialect)
def get_function(self, function_cls):
    """
    Add support for the _method attribute
    """

    try:
        return function_cls._method
    except AttributeError:
        return self.__super__get_function(function_cls)

The monkeypatching functions look like this:

def monkeypatch(*args):
    """
    Decorator to monkeypatch a function into class as a method
    """

    def inner(func):
        name = func.__name__

        for cls in args:
            old = getattr(cls, name)
            setattr(cls, '__super__{}'.format(name), old)

            setattr(cls, name, func)

    return inner


def monkeypatchclass(cls):
    """
    Decorator to monkeypatch a class as a baseclass of @cls
    """

    def inner(basecls):
        cls.__bases__ += (basecls,)

        return basecls

    return inner

Finally we can do queries like this:

>>> track = session.query(Track).get(1)
>>> session.scalar(track.points.length_spheroid('SPHEROID["WGS 84",6378137,298.257223563]'))
6791.87502950043

Code on GitHub.

scratching my own itch [or: why I still love open source]

I’m visiting my parents for the long weekend. Sitting in the airport I decided I should use my spare time to write some documentation, so sitting in the airport was the first time I’d tried to connect to work’s OpenVPN server. While it’s awesome that Network Manager can now import OpenVPN configs, it didn’t work because NM doesn’t support the crucial keysize parameter.

Rather than work around the problem, which some people have done, but would annoyingly break my other OpenVPNs, I used the fact that it’s open source to fix the problem properly.

My Dad asked if I was working. No, well, not really. I’m fixing the interface to my VPN client so I can connect to work’s VPN, I replied. Unglaublich! my father remarked. Not unbelievable, because it’s open source!

Review: GNOME 3 Application Development: Beginner’s Guide

The folk at Packt Publishing sent me an e-copy of GNOME 3 Application Development Beginners Guide a month or so ago.

I’ve been putting off this review because I don’t think this is an very good book and it’s hard to write bad reviews.

First off, the book’s Javascript sections use Seed. I think this is an unconventional choice given that the shell and most of GNOME uses gjs. It had been my experience with the Javascript bindings that gjs was significantly more mature, a view which is confirmed by the fact that Seed has had very little development in the last 18 months.

The book does not seem to use GTK+ best practice, like using Gtk.Grid or Gtk.Application and not using c_new constructor. It is full of things like use of Vala’s [CCode] pragma, but I don’t see why. I felt important and powerful facilities in GLib like properties were not properly explained, especially property binding. There was also a lack of understanding, for example, referring to Timeout objects, which don’t exist (the structure you’re looking for is a Source).

I do like that it uses Anjuta. It’s a shame that it requires unexplained hacks to get things building.

The Clutter section was very poor. Comparing Clutter to GTK+ is simply not reasonable. Clutter is a scene graph API, which doesn’t really have a comparison in the GTK+ stack, which goes from drawing layer to widget layer with no intermediate layer. I immediately noticed the Clutter examples hardcoded layout instead of using a layout manager.

The multimedia section had the user installing non-free codecs. Then it uses alsasink and not auto*sink. It spends a lot of time setting up GStreamer pipelines, rather than using decodebin and playbin, maybe this improves understanding, but I think it mostly will lead to the creation of very rigid apps.

I stopped reading and started skimming at this point. I did again notice weird things like creating JSON using append methods and not the handy JSON-GLib. The examples of HTML5 applications with WebKit perhaps would explain why Seed, except the wrapper is written in Vala, so there’s no problem of conflicting JS engines (I think it works fine anyway, right?). Similarly the application accesses applications by looking in /usr/share/applications rather than libgnome-menus. Again, this will lead to very rigid apps that don’t work very well and doesn’t teach beginners the best practice for GNOME development.

There’s stuff that’s just weird, the system requirements are significantly more powerful than my last computer, on which I was doing GNOME development just fine. There is discussion of how to switch to GNOME Shell, as if it’s required, whereas you can develop GNOME apps in Unity and XFCE just fine.

The typesetting of the book is poor. The source code is weirdly indented and I feel like it lacked readability (there’s great syntax highlighting available for printed text). There are grammatical mistakes that really should have been picked up in editing. The screenshots are blurry in the PDF (looks like some kind of busted bilinear filtering?). Also I can see the resize indicator on the mouse. Generally these serve to make the book look unprofessional.

Finally I don’t think the book really leads the new developer into the community as the best source to get help, which they will undoubtedly need. Of course, the community has already produced some excellent tutorials, which I think new developers would be much better off with.

All up I’m giving it 1 star.

GNOME 3 Application Development: Beginner’s Guide: ★☆☆☆☆