robustness testing

In a recent weird moment in #swfdec, we decided we wanted to do memory allocation robustness testing in Swfdec. Swfdec has some beginnings of a memory management facility, which is similar to malloc returning NULL, but is supposed to be able to live in a glib world: If you need memory, you ask if it is available by calling swfdec_as_context_use_mem. In theory that function returns FALSE if your Flash file used up too much memory. Of course, those failure paths were very untested, because returning FALSE has not been implemented. So what we needed was a function, that would be able to return FALSE at any allocation, just like being able to return NULL at some specific malloc.

I knew there were some projects that do rigorous memory checking in their test suites, so I had a look at how they do it. Apparently, there doesn’t exist a tool for doing that, as everybody has their own solution. Cairo for example uses Chris Wilson’s memfault valgrind skin, which injects malloc failures into running code (comand line options). D-Bus uses evironment variables for practically the same thing:
Both allow you to specify to return NULL from the Nth malloc (with N specified by you). So to get complete coverage of your error handling code, you need to run your program again and again with N increased by one until you have failed for every malloc. That is very slow, as some of the Flash files we ran do 150.000 allocations – and we don’t even do good memory management yet, I expect that to increase by a factor of 10 once we refactor this. And it’s extra slow, if you run it in a valgrind skin. So we needed a faster solution.

The problem that takes so much time is of course easy to identify: You need to run all the code with succeeding malloc calls again and again until you arrive at the failure. What would be a lot faster is being able to return NULL from malloc, see if everything turned out ok, and then continue normally from the malloc. It turns out this is easy. It took 100 lines of code to do this for the above-mentioned swfdec memory function. What I do is forking the process on every malloc. The child returns NULL, the parent continues successfully. That way, you spawn a child for every malloc failure and the child just runs the failure handler. All usual code is only run once – in the parent.

There’s many things that are great using this approach. For a start, it is a lot faster. And not only that, it scales linearly with the amount of allocations done as opposed to the old approach, which was O(N^2). It also provides an easy way to do bookkeeping: the parent reaps all the children and checks if they exited succesfully, otherwise it prints an error message. So you even get a nice output.

5 comments ↓

#1 Chris Lord on 11.03.07 at 15:42

Very neat :) So if a fork of a process segfaults, that doesn’t bring the original process down? I guess that’s the difference between threads and processes…?

#2 pvanhoof on 11.04.07 at 12:37

Neat, clever trick. What about doing this for g_slice_alloc, and g_malloc too? Would this work too?

#3 otte on 11.04.07 at 13:12

Chris: man waitpid
You get all sorts of useful information from that function. The only problem my implementation still has is infinite loops, because the child doesn’t terminate there. But yeah, it’s one nice difference between processes and threads. The other important difference is that the child cannot change global variables in the parent, there’s no shared memory.

Philip: There’s two problems you have with glib in general:
1) The normal glib allocation functions (like g_malloc or g_slice_alloc) abort the program when malloc returns NULL. So it’s pretty useless to do these tests in normal glib-using programs. And this is because
2) The kernel overcommits memory (see http://www.redhat.com/magazine/001nov04/features/vm/ for details). This makes malloc often return a non-null pointer and then the app terminates when you access that memory – especially if you do lots of small allocations, like glib does. That’s why Swfdec uses the approach of tracking the file’s memory itself. That way we know early enough when we run into problems and abort that specific Flash file.

#4 Peter Lund on 11.04.07 at 15:41

Can’t you use ulimit to sort of handle infinite loops?

#5 Andy on 11.05.07 at 09:14

Very clever Mr. Otte.