23.06.2008 Writing Unit Tests with GLib

See: http://blogs.testbit.eu/timj/2008/06/24/23062008-writing-unit-tests-with-glib/ (page moved)

Every other week, someone asks how to use the new unit testing framework in GLib (released with GLib-2.16.0) and Gtk+ (to be released with the next stable). First, here is a write-up from last December that summarizes all important aspects: Test Framework Mini Tutorial.

For people building packages that use the new framework, the following new Makefile rules will be of interest:

  • make test
    Run all tests recursively from $(TEST_PROGS), abort on first error.
  • make test-report
    Run all tests recursively, continue on errors and generate test-report.xml.
  • make perf-report
    Run all tests recursively, enable performance tests (this usually takes significantly longer), continue on errors, generate perf-report.xml.
  • make full-report
    Run all tests recursively, enable performance tests, enable slow (thorough) tests, continue on errors, generate full-report.xml.
  • make check
    Run make test in addition to automake checks.

After GUADEC, we will be looking into getting build machines setup that’ll regularly build GLib, Gtk+ and friends, run the unit testing suites and provide reports online.

For those pondering to write unit tests, but too lazy to look at the tutorial:

  • Implementing a test program is very easy, the only things needed are:
      // initialize test program
      gtk_test_init (&argc, &argv);
      // hook up your test functions
      g_test_add_func ("/Simple Test Case", simple_test_case);
      // run tests from the suite
      return g_test_run();
  • In most cases, a test function can be as simple as:
      static void
      simple_test_case (void)
      {
        // a suitable test
        g_assert (g_bit_storage (1) == 1);
        // a test with verbose error message
        g_assert_cmpint (g_bit_storage (1), ==, 1);
      }

    Tests that abort, e.g. via g_assert() or g_error(), are registered as failing tests with the framework. Also, the gtester utility used to implement the above Makefile rules will restart a test binary after a test function failed and continue to run remaining tests if g_test_add_func() has been used multiple times.

  • Checks in tests can be written with if() and g_error() or exit(1), or simply by using variants of g_assert(). For unit tests in particular, an extended set of assertions has been added, the benefit of using these are the printouts of the involved values when an assertion doesn’t hold:
      g_assert_cmpstr   (stringa, cmpop, stringb);
      g_assert_cmpint   (int64a,  cmpop, int64b);
      g_assert_cmpuint  (uint64a, cmpop, uint64b);
      g_assert_cmphex   (uint64a, cmpop, uint64b);
      g_assert_cmpfloat (doublea, cmpop, doubleb);

    For instance:
    char *string = "foo"; g_assert_cmpstr (string, ==, "bar");
    yields:
    ERROR: assertion failed (string == "bar"): ("foo" == "bar")

  • The framework makes it easy to test program output in unit tests:
      static void
      test_program_output (void)
      {
        if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDOUT |
                                 G_TEST_TRAP_SILENCE_STDERR))
          {
            g_print ("some stdout text: somagic17\n");
            g_printerr ("some stderr text: semagic43\n");
            exit (0);
          }
        g_test_trap_assert_passed();
        g_test_trap_assert_stdout ("*somagic17*");
        g_test_trap_assert_stderr ("*semagic43*");
      }
  • And it is similarly easy to test and verify intentional program abortion:
      static void
      test_fork_fail (void)
      {
        if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDERR))
          {
            g_assert_not_reached();
          }
        g_test_trap_assert_failed();
        g_test_trap_assert_stderr ("*ERROR*test_fork_fail*not*reached*");
      }
  • The above and more tests are showcased in GLib: glib/tests/testing.c.

There’s of course lots of room left to improve GLib and Gtk+ unit tests, and also to improve the current framework. For a speculative, non-comprehensive list, here are some ideas from the unit testing section of my personal TODO:

  • Introduce 2D marker recognition for graphical unit testing of Gtk+ layouts (prototyped in Rapicorn).
  • Provide functionality to determine image similarities to allow for pixel image based unit tests (port this from Rapicorn).
  • Implement state dumps to automate result specification and verification in unit tests. (This will allow us to avoid adding lots of abusable testing hooks to our API.)
  • Integrate performance statistics (like #354457) and other related information into test reports.
  • Publically install a variant of the Makefile.decl file used in Gtk+ to implement the test framework rules and Xvfb swallowing of test programs. This is needed by other projects to run unit tests the way Gtk+ does.
  • Implement the unit test ideas that are outlined at the end of this email: Gtk+ unit tests (brainstorming).