Every other week, someone asks how to use the new unit testing framework in GLib (released with GLib-2.16.0) and Gtk+ (to be released with the next stable). First, here is a write-up from last December that summarizes all important aspects: Test Framework Mini Tutorial.
For people building packages that use the new framework, the following new Makefile rules will be of interest:
-
make test
Run all tests recursively from $(TEST_PROGS), abort on first error. -
make test-report
Run all tests recursively, continue on errors and generatetest-report.xml. -
make perf-report
Run all tests recursively, enable performance tests (this usually takes significantly longer), continue on errors, generateperf-report.xml. -
make full-report
Run all tests recursively, enable performance tests, enable slow (thorough) tests, continue on errors, generatefull-report.xml. -
make check
Run make test in addition to automake checks.
After GUADEC, we will be looking into getting build machines setup that’ll regularly build GLib, Gtk+ and friends, run the unit testing suites and provide reports online.
For those pondering to write unit tests, but too lazy to look at the tutorial:
- Implementing a test program is very easy, the only things needed are:
// initialize test program gtk_test_init (&argc, &argv); // hook up your test functions g_test_add_func ("/Simple Test Case", simple_test_case); // run tests from the suite return g_test_run(); - In most cases, a test function can be as simple as:
static void simple_test_case (void) { // a suitable test g_assert (g_bit_storage (1) == 1); // a test with verbose error message g_assert_cmpint (g_bit_storage (1), ==, 1); }Tests that abort, e.g. via g_assert() or g_error(), are registered as failing tests with the framework. Also, the gtester utility used to implement the above Makefile rules will restart a test binary after a test function failed and continue to run remaining tests if g_test_add_func() has been used multiple times.
- Checks in tests can be written with if() and g_error() or exit(1), or simply by using variants of g_assert(). For unit tests in particular, an extended set of assertions has been added, the benefit of using these are the printouts of the involved values when an assertion doesn’t hold:
g_assert_cmpstr (stringa, cmpop, stringb); g_assert_cmpint (int64a, cmpop, int64b); g_assert_cmpuint (uint64a, cmpop, uint64b); g_assert_cmphex (uint64a, cmpop, uint64b); g_assert_cmpfloat (doublea, cmpop, doubleb);
For instance:
char *string = "foo"; g_assert_cmpstr (string, ==, "bar");
yields:
ERROR: assertion failed (string == "bar"): ("foo" == "bar") - The framework makes it easy to test program output in unit tests:
static void test_program_output (void) { if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDOUT | G_TEST_TRAP_SILENCE_STDERR)) { g_print ("some stdout text: somagic17\n"); g_printerr ("some stderr text: semagic43\n"); exit (0); } g_test_trap_assert_passed(); g_test_trap_assert_stdout ("*somagic17*"); g_test_trap_assert_stderr ("*semagic43*"); } - And it is similarly easy to test and verify intentional program abortion:
static void test_fork_fail (void) { if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDERR)) { g_assert_not_reached(); } g_test_trap_assert_failed(); g_test_trap_assert_stderr ("*ERROR*test_fork_fail*not*reached*"); } - The above and more tests are showcased in GLib: glib/tests/testing.c.
There’s of course lots of room left to improve GLib and Gtk+ unit tests, and also to improve the current framework. For a speculative, non-comprehensive list, here are some ideas from the unit testing section of my personal TODO:
- Introduce 2D marker recognition for graphical unit testing of Gtk+ layouts (prototyped in Rapicorn).
- Provide functionality to determine image similarities to allow for pixel image based unit tests (port this from Rapicorn).
- Implement state dumps to automate result specification and verification in unit tests. (This will allow us to avoid adding lots of abusable testing hooks to our API.)
- Integrate performance statistics (like #354457) and other related information into test reports.
- Publically install a variant of the Makefile.decl file used in Gtk+ to implement the test framework rules and Xvfb swallowing of test programs. This is needed by other projects to run unit tests the way Gtk+ does.
- Implement the unit test ideas that are outlined at the end of this email: Gtk+ unit tests (brainstorming).
