Category: Gtk+

See: http://blogs.testbit.eu/timj/2008/06/24/23062008-writing-unit-tests-with-glib/ (page moved)

Every other week, someone asks how to use the new unit testing framework in GLib (released with GLib-2.16.0) and Gtk+ (to be released with the next stable). First, here is a write-up from last December that summarizes all important aspects: Test Framework Mini Tutorial.

For people building packages that use the new framework, the following new Makefile rules will be of interest:

• make test
  Run all tests recursively from $(TEST_PROGS), abort on first error.
• make test-report
  Run all tests recursively, continue on errors and generate test-report.xml.
• make perf-report
  Run all tests recursively, enable performance tests (this usually takes significantly longer), continue on errors, generate perf-report.xml.
• make full-report
  Run all tests recursively, enable performance tests, enable slow (thorough) tests, continue on errors, generate full-report.xml.
• make check
  Run make test in addition to automake checks.

After GUADEC, we will be looking into getting build machines setup that’ll regularly build GLib, Gtk+ and friends, run the unit testing suites and provide reports online.

For those pondering to write unit tests, but too lazy to look at the tutorial:

• Implementing a test program is very easy, the only things needed are:

    // initialize test program
    gtk_test_init (&argc, &argv);
    // hook up your test functions
    g_test_add_func ("/Simple Test Case", simple_test_case);
    // run tests from the suite
    return g_test_run();

• In most cases, a test function can be as simple as:

    static void
    simple_test_case (void)
    {
      // a suitable test
      g_assert (g_bit_storage (1) == 1);
      // a test with verbose error message
      g_assert_cmpint (g_bit_storage (1), ==, 1);
    }

  Tests that abort, e.g. via g_assert() or g_error(), are registered as failing tests with the framework. Also, the gtester utility used to implement the above Makefile rules will restart a test binary after a test function failed and continue to run remaining tests if g_test_add_func() has been used multiple times.

• Checks in tests can be written with if() and g_error() or exit(1), or simply by using variants of g_assert(). For unit tests in particular, an extended set of assertions has been added, the benefit of using these are the printouts of the involved values when an assertion doesn’t hold:

    g_assert_cmpstr   (stringa, cmpop, stringb);
    g_assert_cmpint   (int64a,  cmpop, int64b);
    g_assert_cmpuint  (uint64a, cmpop, uint64b);
    g_assert_cmphex   (uint64a, cmpop, uint64b);
    g_assert_cmpfloat (doublea, cmpop, doubleb);

  For instance:
  char *string = "foo"; g_assert_cmpstr (string, ==, "bar");
  yields:
  ERROR: assertion failed (string == "bar"): ("foo" == "bar")

• The framework makes it easy to test program output in unit tests:

    static void
    test_program_output (void)
    {
      if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDOUT |
                               G_TEST_TRAP_SILENCE_STDERR))
        {
          g_print ("some stdout text: somagic17\n");
          g_printerr ("some stderr text: semagic43\n");
          exit (0);
        }
      g_test_trap_assert_passed();
      g_test_trap_assert_stdout ("*somagic17*");
      g_test_trap_assert_stderr ("*semagic43*");
    }

• And it is similarly easy to test and verify intentional program abortion:

    static void
    test_fork_fail (void)
    {
      if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDERR))
        {
          g_assert_not_reached();
        }
      g_test_trap_assert_failed();
      g_test_trap_assert_stderr ("*ERROR*test_fork_fail*not*reached*");
    }

• The above and more tests are showcased in GLib: glib/tests/testing.c.

There’s of course lots of room left to improve GLib and Gtk+ unit tests, and also to improve the current framework. For a speculative, non-comprehensive list, here are some ideas from the unit testing section of my personal TODO:

• Introduce 2D marker recognition for graphical unit testing of Gtk+ layouts (prototyped in Rapicorn).
• Provide functionality to determine image similarities to allow for pixel image based unit tests (port this from Rapicorn).
• Implement state dumps to automate result specification and verification in unit tests. (This will allow us to avoid adding lots of abusable testing hooks to our API.)
• Integrate performance statistics (like #354457) and other related information into test reports.
• Publically install a variant of the Makefile.decl file used in Gtk+ to implement the test framework rules and Xvfb swallowing of test programs. This is needed by other projects to run unit tests the way Gtk+ does.
• Implement the unit test ideas that are outlined at the end of this email: Gtk+ unit tests (brainstorming).