Category: Uncategorized

So I found a bit of time to hack on my project today. Today’s task was to load and validate data coming from Geoscience Australia’s SRTM digital elevation model data, which I downloaded from their elevation data portal last week. ((Note to the unwary, seems to be buggy in Chrome?)) The data is Creative Commons, so I might just upload it somewhere, if I can find a place for 2GB of elevation data.

This let me load elevation data in a 3 arcsecond (about 100m) grid, which I did using the ubiquitous GDAL via its Python API. Initial code is here. It doesn’t do anything super clever yet, like check and normalise the projection, because I don’t need to. ((I did write a skeleton context manager for gdal.Open. I say skeleton because it doesn’t actually do anything smart like turning errors from GDAL into Exceptions, because I didn’t have any errors to handle.))

Looking at plots of values can give you a gist of what’s what (oh look, it goes out to sea, and then the data is masked out) but it doesn’t really validate anything. I could do validation runs against my GPS tracks, but for a first pass, I decided it would be easier to validate using Google’s Elevation API. This is a pretty neat web service that you make a request to, and it gives you back some JSON (or XML). There are undoubtedly Python APIs to access this, but it’s pretty easy to do a simple call with urllib2 or httplib. I chose to reuse my httplib Client wrapper from my RunKeeper/HealthGraph API. I wrote it directly in the test.

For a real unit test, I would have probably calculated the residuals, and ensured they sat within some acceptable range, but I’m lazy, so instead I just plotted them together. Google’s data, you will notice, includes bathymetry, which is actually pretty neat.

So as I’ve spent the last year learning Melbourne as a cyclist, there’s been a few times where I’ve found myself on an absolutely staggering hill, only to go down it again, and worse find there was another way that totally avoided the hill; or I’ve chosen routes that subject me to staggering head winds only to be told I should have taken another route instead.

This got me thinking, with everyone tracking their cycles on their smartphones, why couldn’t I feed all of this data into a model, along with some data like NASA’s elevation grids, or the Bureau of Meteorology’s wind observations. As something to keep me occupied over Christmas, I started a little project on the plane to Perth.

It turns out RunKeeper has this handy API that lets you access everything stored there. Unfortunately it seems that no one has really written a good Python API for this, so I put one together.

Throw in a bit of NumPy, PyProj (to convert to rectilinear coordinates) and Matplotlib (to plot it) and you can get a graph that looks like this (which thankfully looks a lot like RunKeeper’s graph):

If we do some long window smoothing, we can get an idea of a cyclist’s average speed and then calculate a percentage anomaly from this average speed. This lets us compensate for different cyclists, how tired they are or if they’re riding with someone else ((This does have the side effect of reducing the signal from head/tail winds, especially on straight trips, I need to think about this more.)).

If we then do this for lots of tracks and grid the results based on whether the velocity vector at each point is headed towards or away from Melbourne ((We need this, otherwise the velocity anomaly would average out depending on which direction we’re headed up/down a hill.)) we can get spatial plots that look like this (blue is -1 and red is 1):

If you squint at the graphs you can sort of see that there are many places where the blue/red are inverted, which is promising, it meant something was making us faster one way and slower the other (a hill or wind or the pub). You can also see that I still don’t really have enough data, I tend to always cycle the same routes. If I want to start considering factors that are highly temporally variable, like wind, I’m going to need a lot more data to keep the number of datapoints (always want to call this fold) high in my temporal bins.

The next step I suppose is to set up the RunKeeper download as a web service, so people can submit their RunKeeper tracks to me. This means I’m going to have to fix up some hard coded assumptions in the code, like the UTM zone for rectilinear projection, and what constitutes an inbound or an outbound route. Unsurprisingly this has become a lot more ambitious than a summer project.

If you feel like having a play, there is source code.

October 16th is Ada Lovelace Day. A day that showcases women in engineering, maths, science and technology by profiling a woman technologist, scientist, engineer or mathematician on your blog.

This year I’m writing about Elaine Miles, a researcher at the Australian Bureau of Meteorology, who I met through mutual colleagues over lunch one day. She has since become one of my go-to people whenever I require a crash-course in something. She awesomely let me interview her for Ada Lovelace Day.

Miles is a physicist working at the Centre for Australian Weather at Climate Research (CAWCR), a joint project between the Bureau of Meteorology and the Commonwealth Scientific and Industrial Research Organisation (CSIRO), where she is investigating the use of dynamic models to predict sea level in the Western Pacific.

Miles studied Applied Mathematics and Physics at the University of Melbourne. She attributes her love of maths to primary school, where she recalls being chastised by her teacher for attempting the subtraction problems further on in the workbook before they had been taught subtraction. She says she would always lament when another class ran over and cut into the maths lesson. Her love of maths originally led her to enroll in Electrical Engineering, but she didn’t like the black box thinking that engineering encourages, preferring to understand concepts from first principles.

Completing a Bachelor in Applied Mathematics, she went on to do Honours in Physics, working on a project in art conservation, which it turns out is an extremely technical field. She built a laser interferometer using off-the-shelf parts (laser, CCD camera and a laptop) to monitor canvas artworks and detect the problems caused to art by changes in microclimate.

After teaching English in Japan for a year, Miles returned to Melbourne where she began her PhD (Miles is not related to Dr Elaine Miles the glass artist). Miles says she wanted to be learning or developing new things (plus she had unfinished business in art conservation) and so a PhD was the logical progression. Her PhD focused on two areas: the science of paint drying (literally watching paint dry she says) and subsurfacing imaging. She had a focus on south-east Asia, where Western art production techniques are prevalent but unsuitable because of the different climate.

Miles spent 3 months working with galleries in the Philippines where she used her own laser speckle interferometers to study artwork hanging in the gallery, in-situ. As far as she’s aware, studying art in-situ had never been done before. This work allows conservators to determine best practice and a course of action for storing and restoring works of art.

With her PhD close to being submitted, Miles began work at CAWCR where she first worked on data assimilation of weather balloon observations into weather forecasting models. She then moved on to verifying rainfall prediction models and getting weather radar data assimilated into the model.

For the last 10 months she has worked with the Pacific-Australia Climate Change Science and Adaptation Planning Program (PACCSAPP), where she investigates applying POAMA (Predictive Ocean Atmosphere Model for Australia), a dynamic, coupled ocean-atmospheric, multi-model ensemble global seasonal prediction model, to forecast global sea level anomalies 1-9 months in the future, specifically validating predictions with observations in the Western Pacific. This is the first time dynamic models have been used to predict medium-term sea level, and forms an extremely important part of helping the Pacific adapt to the immediate effects of climate change.

As for the future, Miles looks forward to getting her PhD submitted, but would like to continue working with sea level modelling. She hopes to start leading projects in Australia and around the world.

A friend of mine recently suggested people donate to the Women’s Domestic Violence Crisis Service, whose website provides a quick escape button (like a boss button) for victims of abuse to get away from the site quickly. Unfortunately the button simply takes you to Google (via an image map, I wonder why?). It doesn’t manipulate your history.

For people in abusive relationships, leaving behind browser history saying they were accessing a site to get help could be dangerous (this is why there’s a quick escape button in the first place).

HTML5 includes a History API, which will let you manipulate the most recent entry, and you could just screw up all the site history, that would be annoying for other people. It seems like the correct answer here is private browsing or incognito mode. While the warning page could include instructions on how to activate incognito mode for your browser (and offer to screw up the history) one wonders why HTML5 doesn’t include a method to load a site private/incognito so websites could offer to stay out of your browser history? I’m sure it would prove very popular with porn sites, but it might also help to protect some people.

Thanks everyone at GUADEC for the sweet thoughts. Miss you all. Hope I’ll see you next year 🙂

I was hit by a car on my cycle to work on Tuesday morning. Broke my collarbone, requiring surgery to repair. I’m okay. Have a couple of weeks off work. Full write up on my other blog (warning: contains details maybe not for the squeamish).

Had to import a repository from Mercurial to Git today.

I used fast-export:

git clone git://repo.or.cz/fast-export.git

Because the author fields were an utter mess, I needed to create an authors map. This is not well documented but it’s of the form email address to author <email>. For example:

Administrator@MACHINE.localdomain=Bob McBadgers <bob@badgers.st>
badgerbob@gmail.com=Bob McBadgers <bob@badgers.st>
badgerbob@MyMachine=Bob McBadgers <bob@badgers.st>

You then create a new git repo, e.g.

mkdir myapp.git ; cd myapp.git
git init --bare --shared=group
../fast-export/hg-fast-export.sh -A ../authors.map -r ../myapp.hg/

If it produces an empty repo, ensure you have the Mercurial Python modules available.

Just got back from a pretty amazing two week holiday visiting my best friend Steph in Beijing (read about it here).

Tomorrow I start my new job as a Senior IT Officer at the Bureau of Meteorology. I will be working on a foreign aid project to produce climate forecasting software for pacific nations. This should be exciting work with lots of Linux and open source.

I’m a little bit nervous, I haven’t worked in an office for almost four years. Spent some time this weekend shopping for new work clothes and appropriate shoes. Apparently civil servants don’t wear jeans and kicks to work.

This is my last day at Collabora.

It’s more than a little sad. I’ve been at Collabora now since the start of 2009. I’ve watched the company grow from something small into something impressive. Collabora is the company for open source consultancy. I’ve been to some great events and met some great people. Some of whom are now good friends. I am going to miss visiting Cambridge.

Leaving was a personal decision. I am a social person, and working from home for so many years has left me craving an office environment again. It’s weird to leave somewhere when you work from home. There’s no cake, you don’t pack up your desk.

I want to thank Collabora for all the conferences it has let me attend, and all the open source work it has let me do just because it was cool.

There is a next. First I am taking two weeks holiday in Beijing, where I will be visiting my bestie and helping her celebrate her birthday. After that I will be starting work on a foreign aid project with the Australian federal government.

I am planning on staying involved with GNOME, although I’m not sure in what capacity. Unfortunately I won’t make it to GUADEC this year. I’m doing so much other flying in July and August that the idea of a 30h flight to Spain makes me feel a little ill. I will be at Pycon AU and almost definitely at linux.conf.au, hopefully at GUADEC next year (I know, I said that last year).

I leave you with this photo of Kings College, taken out of the window of the old Collabora offices:

Yesterday Michael Meeks expressed his distaste at GUADEC’s Attendee’s policy.

I have personally advocated for conferences to adopt anti-harassment guidelines. I am also an advisor to The Ada Initiative, which is involved with this work.

Michael writes:

Fair enough getting aggressive against stalking, groping and such horrors; but encouraging censorship of “offensive” verbal comments related to sexual orientation, religion etc. looks like a persecutors charter in the making. What is offensive ? and to whom ? the fear being that -very- quickly such good aspirations slide from “applied common sense” into a militant denial of a basic right to reasonably critique others’ world-views. Put another way I’m really happy for people to tell me how wrong-headed I am on any number of engaging topics, and to discuss them in an animated and friendly fashion. I loathe a framework that will discourage people from coming and saying: “your Christian faith seems incomprehensibly stupid to me” (for example), or “the crazy English always fall down the stairs”, or whatever.

Here is the relevant section of the attendee’s policy:

Harassment includes offensive verbal comments related to gender, sexual orientation, disability, physical appearance, body size, race, religion, sexual images in public spaces, deliberate intimidation, stalking, following, unauthorized or inappropriate photography or recording, sustained disruption of talks or other events, inappropriate physical contact, and unwelcome sexual attention.

Firstly, if you can’t understand the point of an anti-harassment policy, or you think what’s written is obvious, then the policy probably isn’t there to protect you. Typically, when you are a member of the dominant demographic group in a space, you do not need to be protected in this way. On the other hand, I’ve heard women be told you don’t belong here, that they’re unfuckable or a dyke. I’ve been followed, I’ve had my picture taken repeatedly without my consent, one time including a sleazy remark. My friends have been stalked, photographed discreetly, inappropriately touched and sexually assaulted.

More often than not the attempted defence against this sort of behaviour is I didn’t know. Experience shows that some people do need acceptable and unacceptable behavior spelled out precisely.

Offence is in the ear of the listener (who is not always the recipient of the comment). It’s also important to distinguish the ability to critique from being offensive. Dialogue is negotiated. You can willingly consent to discuss your religion, but you don’t have to accept being verbally abused because of it. The list here indicates the common problem areas across the broad technical community (GUADEC/Desktop Summit is better than most, but not immune). Most of these are also present in various jurisdictions’ anti-discrimination law.

The goal of these policies is to make sure that everyone at the event has a good time. That no one feels like less of a person or has their day ruined because someone else was nasty to them because they’re Muslim, gay, African, obese, partially-deaf or so forth. That no one should ever feel unsafe. The goal is not to stifle discussion or censor fun (if you think hurting people is fun, consider therapy). There are many ways of being funny without putting your audience down.

People should feel comfortable and safe in our community. An anti-harassment policy is a statement that our community makes an effort to be inclusive, friendly and safe for everyone.

For more information, see Conference anti-harassment policy resources on the Geek Feminism wiki.