rendering cleanup

I’ve spent the last 2 weeks in a GTK branch purging the old X drawing code from GDK to see if it would work. Not only did it work, but it made quite a few people happy to see this done. So this will probably be merged into GTK master rather soon, along with lots of deprecations for GTK 2.22. The patchset removes roughly 150 API calls from the GTK code. If you have an application or library that doesn’t use Cairo yet but still uses the old drawing calls, you will have to port it to Cairo to be GTK compatible. What follows is a rough draft of what I intend to become a porting guide for the GTK documentation.

What code was removed?

The cleanup basically just removes two objects from GDK: GdkGC and GdkImage. All the functions using them, most prominently the gdk_draw_ set of functions like gdk_draw_rectangle() and gdk_draw_drawable() were of course removed, too. As GdkGC is roughly equivalent to cairo_t and GdkImage was used for drawing images to GdkDrawables, which Cairo supports automatically, a transition is usually straightforward and does not require refactoring.

Why was this cleanup done?

  • Bad semantics
    In the X11 backend, a GdkGC is represented by a server-side object so creation and destruction are very expensive. This resulted in GTK creating these objects in advance and requiring code to use a “setup, use, restore” approach, which required the extra cleanup work and was error-prone. Also, a lot of operations provided by GdkGC were hard to understand and quirky. This posed a lot of issues when trying to port GTK to other platforms like Microsoft Windows, Apple’s Quartz rendering engine or OpenGL.

  • Outdated featureset
    The GDK drawing API mapped very closely to the core X drawing API which was invented 20 years ago while the Cairo API is modeled after the modern PDF rendering model, which supports bezier curves, antialiasing, translucent drawables, antialiased clipping and masking and resolution independence, all of which are not available in the GDK API.

  • Bad API
    A lot of functions in GDK that took a GdkGC as an argument used only some of the GC’s properties but not all of them. Unfortunately it was rarely clear what operations were supported and which weren’t.

  • Performance
    A lot of work is spent on making Cairo accelerated better on GPUs, while such work is not done for the old GdkGC code. In a lot of cases trying to this was not even possible or very complex due to the mentioned quirkiness of the old drawing operations.

When and how should you port your code?

Porting your code should happen as soon as possible. The replacement APIs have been available since Gtk 2.8, so it is not necessary to depend on recent code.
When porting your code, you want to replace GdkImage uses with a Cairo image surface and draw to/from it using the usual Cairo APIs. Whenever your code uses a GdkGC, you want to replace it by creating a cairo_t instead and replace each function call with the equivalent Cairo calls. See the outdated function’s Gtk 2 documentation for the replacements.

A few examples

This section contains examples for a few common idioms used by applications that have been ported to use Cairo and how the code was replaced.

  • Drawing a GdkPixbuf onto a GdkDrawable
    Drawing a pixbuf onto a drawable used to be done like this:

    gdk_draw_pixbuf (window,
                     gtk_widget_get_style (widget)->black_gc,
                     pixbuf,
                     0, 0
                     x, y,
                     gdk_pixbuf_get_width (pixbuf),
                     gdk_pixbuf_get_height (pixbuf),
                     GDK_RGB_DITHER_NORMAL,
                     0, 0);

    Doing the same thing with Cairo:

    cairo_t *cr = gdk_cairo_create (window);
    gdk_cairo_set_source_pixbuf (cr, pixbuf, x, y);
    cairo_paint (cr);
    cairo_destroy (cr);

    Note that very similar code can be used for drawing pixmaps by using gdk_cairo_set_source_pixmap() instead of gdk_cairo_set_source_pixbuf().

  • Drawing a tiled GdkPixmap to a GdkDrawable
    Tiled pixmaps are often used for drawing backgrounds. Old code looked something like this:

    GdkGCValues gc_values;
    GdkGC *gc;
    
    /* setup */
    gc = gtk_widget_get_style (widget)->black_gc;
    gdk_gc_set_tile (gc, pixmap);
    gdk_gc_set_fill (gc, GDK_TILED);
    gdk_gc_set_ts_origin (gc, x_origin, y_origin);
    /* use */
    gdk_draw_rectangle (drawable, gc, TRUE, 0, 0, width, height);
    /* restore */
    gdk_gc_set_tile (gc, NULL);
    gdk_gc_set_fill (gc, GDK_SOLID);
    gdk_gc_set_ts_origin (gc, 0, 0);

    The equivalent Cairo code looks like this:

    cairo_t *cr;
    
    cr = gdk_cairo_create (drawable);
    gdk_cairo_set_source_pixmap (cr, pixmap, x_origin, y_origin);
    cairo_pattern_set_extend (cairo_get_source (cr), CAIRO_EXTEND_REPEAT);
    cairo_rectangle (cr, 0, 0, width, height);
    cairo_fill (cr);
    cairo_destroy (cr);

    Again, you can exchange pixbufs and pixmaps by using gdk_cairo_set_source_pixbuf() instead of gdk_cairo_set_source_pixmap().

  • Drawing a PangoLayout to a clipped area
    Drawing layouts clipped is often used to avoid overdraw or to allow drawing selections. Code would have looked like this:

    GdkGC *gc;
    
    /* setup */
    gc = gtk_widget_get_style (widget)->text_gc[state];
    gdk_gc_set_clip_rectangle (gc, &area);
    /* use */
    gdk_draw_layout (drawable, gc, x, y, layout);
    /* restore */
    gdk_gc_set_clip_rectangle (gc, NULL);

    With Cairo, the same effect can be achieved using:

    cairo_t *cr;
    
    cr = gdk_cairo_create (drawable);
    /* clip */
    gdk_cairo_rectangle (cr, &area);
    cairo_clip (cr);
    /* set the correct source color */
    gdk_cairo_set_source_color (cr, &gtk_widget_get_style (widget)->text[state]);
    /* draw the text */
    cairo_move_to (cr, x, y);
    pango_cairo_show_layout (cr, layout);
    cairo_destroy (cr);

    Clipping using cairo_clip() is of course not restricted to text rendering and can be used everywhere where GC clips were used. And using gdk_cairo_set_source_color() with style colors should be used in all the places where a style’s GC was used to achieve a particular color.

    What should you be aware of?

  • No more stippling
    Stippling is the usage of a bi-level mask, called a GdkBitmap. It was often used to achieve a checkerboard effect. You can use cairo_mask() to achieve this effect. To get a checkerbox mask, you can use code like this:

    static cairo_pattern_t *
    gtk_color_button_get_checkered (void)
    {
        /* need to respect pixman's stride being a multiple of 4 */
        static unsigned char data[8] = { 0xFF, 0x00, 0x00, 0x00,
                                         0x00, 0xFF, 0x00, 0x00 };
        cairo_surface_t *surface;
        cairo_pattern_t *pattern;
    
        surface = cairo_image_surface_create_for_data (data,
                                                       CAIRO_FORMAT_A8,
                                                       2, 2,
                                                       4);
        pattern = cairo_pattern_create_for_surface (surface);
        cairo_surface_destroy (surface);
        cairo_pattern_set_extend (pattern, CAIRO_EXTEND_REPEAT);
        cairo_pattern_set_filter (pattern, CAIRO_FILTER_NEAREST);
    
        return pattern;
    }

    Note that all properties that made use of stippling have been removed from GTK APIs as the result of stippling is not used in practice as it looks very outdated. Most prominently, stippling is absent from text rendering, in particular GtkTextTag.

  • Using the the target drawable also as the source or mask
    gdk_draw_drawable() allowed using the same drawable as source and target. This
    was often used to achieve a scrolling effect. Cairo does not allow this yet.
    You can however use cairo_push_group() to get a different intermediate target
    that you can copy to. So you can replace this code:

    gdk_draw_drawable (pixmap,
                       gc,
                       pixmap,
                       area.x + dx, area.y + dy,
                       area.x, area.y,
                       area.width, area.height);

    By using this code:

    cairo_t *cr = gdk_cairo_create (pixmap);
    /* clipping restricts the intermediate surface's size, so it's a good idea
     * to use it. */
    gdk_cairo_rectangle (cr, &area);
    cairo_clip (cr);
    /* Now push a group to change the target */
    cairo_push_group (cr);
    gdk_cairo_set_source_pixmap (cr, pixmap, dx, dy);
    cairo_paint (cr);
    /* Now copy the intermediate target back */
    cairo_pop_group_to_source (cr);
    cairo_paint (cr);
    cairo_destroy (cr);

    Cairo developers plan to add self-copies in the future to allow exactly this effect, so you might want to keep up on Cairo development to be able to change your code.

  • Using pango_cairo_show_layout() instead of gdk_draw_layout_with_colors()
    GDK provided a way to ignore the color attributes of text and use a hardcoded text color with the gdk_draw_layout_with_colors() function. This is often used to draw text shadows or selections. Pango’s Cairo support does not yet provide this functionality. If you use Pango layouts that change colors, the easiest way to achieve a similar effect is using pango_cairo_layout_path() and cairo_fill() instead of gdk_draw_layout_with_colors(). Note that this results in a slightly uglier-looking text, as subpixel anti-aliasing is not supported.

fun with benchmarks

Disclaimer: Dear Phoronix, this post is totally for you.

First, a graph

comparison graph for xlib on intel
This graph is a comparison of relative performance of the xlib backend on various computers with an Intel GPU. What it shows is the relative performance of using the X server compared to just using the computer’s CPU for rendering the real-world benchmarks we Cairo people compiled. The red bar in the center is the time the CPU renderer takes. If the bar goes to the left, using the X server is slower, if it goes to the right, using the X server is faster. Colors represent the different computers the benchmark ran on (see the legend on the left side), and the numbers tell you how much slower or faster the test ran. So the first number “-3.5″ means that the test ran 3.5x slower using the X server than it ran just using the CPU. And the graph shows very impressively that the GPU is somewhere between 13 times slower and 5 times faster than the CPU depending on the task you do. And even for the exact same task, it can be between 7 times slower and 2 times faster depending on the computer it’s running on. So should you use the GPU for what you do?

Well, what’s actually benchmarked here?

The benchmarks are so-called traces recorded using cairo-trace APPLICATION. This tool will be available with Cairo 1.10. It works a bit like strace in that it intercepts all cairo calls of your application. It then records them to a file and that file can later be played back for benchmarking rendering performance. We’ve so far taken various test cases we considered important and added them to our test-suite. They contain everything from watching a Video on Youtube (swfdec-youtube) over running Mozilla’s Talos testsuite to scrolling in gnome-terminal. The cairo-traces README explains them all. The README also explains how to contribute traces. And we’d love to have more useful traces.

But GL!

Yes, OpenGL is apparently the solution to all problems with rendering performance. Which is why I worked hard in recent weeks to improve Cairo’s GL backend. The same graph as above, just using GL instead of Xlib looks like this:
comparison graph for gl on intel
What do we learn? Yes, it’s equally confusing, from 30 times slower to 3 times faster. And we have twice as fast and 1.5x slower on the same test depending on hardware. So it’s the same mess all over again. We cannot even say if using the CPU is better or worse than using the GPU on the same machine. Take the EeePC results for the firefox-planet-gnome test: Xlib 2.5x faster, GL is 4.5x slower. Yay.

But that’s only Intel graphics

Yeah, we’re all working on Intel chips in Cairo development. Which might be related to the fact that we all either work for Intel or get laptops with Intel GPUs. Fortunately, Chris had a Netbook with an nvidia chip where he ran the tests on – once with the nvidia binary driver and once with the Open Source nouveau driver. Here’s the Xlib results:
comparison graph for xlib on nvidia
Right, that looks as unconclusive as before. Maybe a tad worse than on Intel machines. And nouveau seems to be faster than the binary nvidia driver. Good work! But surely for GL nvidia GPUs will rule the world, in particular with the nvidia binary driver. Let’s have a look:
comparison graph for gl on nvidia
Impressive, isn’t it? GL on nvidia is slow. There’s not a single test where using the CPU would be noticably slower. And this is on a Netbook, where CPUs are slow. And the common knowledge that the binary drivers are better than nouveau aren’t true either: nouveau is significantly better. But then, we had a good laugh because we convinced ourselves that Intel GPUs seem to be a far better choice than nVidia GPUs currently.

That’s not enough data!

Now here’s the problem: To have a really really good laugh, we need to be sure that Intel is better than nVidia and ATI. And we don’t have enough data points to prove that conclusively yet. So if you have a bunch of spare CPU cycles and are not scared of building development code, you could run these benchmarks on your machine and send them to me. That way, you help the Cairo team getting a better overview of where we stand and a good idea of what areas need improvement the most. Here’s what you need to do:

  1. Make sure your distro is up to date. Recent releases – like Fedora 13 or Ubuntu Lucid – are necessary. If you run development trees of everything, that’s even better.
  2. Start downloading http://people.freedesktop.org/~company/stuff/cairo-traces.tar.bz2 – it’s a 103MB file that contains the benchmark traces from the cairo-traces repository.
  3. Install the development backages required to build Cairo. There’s commands like apt-get build-dep cairo or yum-builddep cairo that do that.
  4. Install development packages for OpenGL.
  5. Check out current Cairo from git using git clone git://anongit.freedesktop.org/cairo
  6. Run ./autogen.sh --enable-gl in the checked out cairo dir. In the summary output at the end, ensure that the Xlib and the GL backends will be built.
  7. Run make
  8. change into the perf/ directory.
  9. Run make cairo-perf-trace
  10. Unpack the cairo-traces.tar.bz2 that hopefully finished downloading by now using tar xjf cairo-traces.tar.bz2. This will create a benchmark directory containing the traces.
  11. Run CAIRO_TEST_TARGET=image,xlib,gl ./cairo-perf-trace -r -v -i3 benchmark/* > `git describe`-`date '+%Y%m%d'`.`hostname`.perf. Ideally, you don’t want to use the computer while the benchmark runs. It will output statistics to the terminal while it runs and generate a .perf file, named something like “1.9.8-56-g1373675-20100624.lvs.perf”.
  12. Attach the file into an email and send it to otte@gnome.org, preferrably with information on your CPU, GPU and which distribution and drivers you were running the test on.

Note: The test runs will take a while. Depending on your drivers and CPU, it’ll take between 15 minutes and a day – it takes ~half an hour on my 6months old X200 laptop.

What’s going to happen with all the benchmarks?

Depending on the feedback I get for this, I intend to either weep into a pillow because noone took the time to run the benchmarks, write another blog post with pretty graphics or even do a presentation about it at Linux conferences. Because most of all I intend to make us developers learn from it so that we can improve drivers and libraries so that the tests are 10x faster when you upgrade your distro the next time.

Note: I originally read the nvidia graph wrong and assumed the binary drivers were faster for OpenGL. I double-checked my data: This is not true. nouveau is faster for the Cairo workloads than the nvidia driver.

Youtube out of the box

With today’s GStreamer update, you are able to watch WebM videos. That means you can enjoy Youtube with a stock Fedora 13. You need the following ingredients:

  1. The updates-testing repository enabled until the update hits the stable repository.
  2. A browser that uses GStreamer, such as Epiphany or Midori.
  3. Opting in to the Youtube HTML5 beta.
  4. A Youtube video that is already provided in WebM, such as this one.

This update will also arrive in Fedora 12.

And because quite a few people have asked about this in bug reports or email, I will note that this will be the last feature update I intend to do for GStreamer packages on Fedora 12. As GStreamer updates are quite intrusive and can have unexpected consequences, I don’t like to push new packages into releases unless I run them myself and can easily debug them. So if you want up-to-date GStreamer packages, you will always have to run the latest Fedora release – or of course any non-released alpha, beta or rawhide.

Whose fault is it?

Something I realized today:
The iPad doesn’t play Flash and according to its fans, it’s the fault of the web sites.
Linux doesn’t play Flash and according to its fans, it’s the fault of Linux.

Why is that? Are Linux fans no real fans?

hacking spree

I’ve spent time the last few months looking at OpenGL trying to grok how it works. And I wasn’t very successful. That was until I got 2 epiphanies. The first one is that the OpenGL API really is as bad as it seems. I had always thought OpenGL would be the best thing since sliced bread because everyone claims it’s awesome, from the early days up to recent times, where people claimedd it’ll be the future of graphics on Linux. So I somehow had higher expectations but it seems we’ll just switch from one ugly API to another one. (I’ll not get into ranting about why OpenGL is bad now.)

The other epiphany was related to glEnable (GL_BLEND) and some coce not calling it. Yes, that enables rather important functionality and yes, it’s off by default and no, the lib I was playing with did not enable it. After fixing that, I was able to have OpenGL-generated content in GStreamer pipelines. This video was generated using gst-launch:

The other thing I spent time on was making sure Cairo honours the thread safety promises it wants to give people in the future – ideally from version 1.10 onwards, but it looks like not all backends will achieve that. Before listing the promises, I’ll introduce a new object type in Cairo, the cairo_device_t. A device encapsulates the connection to the rendering backend and is roughly equivalent to a GdkDisplay. As we all know, this is usually not threadsafe by itself. But Cairo wants it to be. So it promises:

  1. Every surface can be used as a source in as many threads as one wishes.

  2. Any surface can be used as a destination in one thread at a time.
  3. Multiple surfaces sharing the same device can be used in multiple threads at the same time.
  4. Cairo rendering will not interfere while developers acquired the device and use it themselves.

Chris had spent time previously on ensuring that the image and xcb backends were threadsafe. I made sure the xlib and GL backends are now threadsafe, too. So you can basically start writing multithreaded applications using Cairo that use hardware accelerated rendering.

As my gst-plugins-cairo plugins are holding up pretty well stability-wise to the GStreamer pipelines I throw at it, I’ve managed to get gst-plugins-gl developers excited. Yay for more contributors! Now we “just” need to get all my patches merged and work on some performance warts and the future can be OpenGL.

3 months

I’ve been in my new job for 90 days now, as some internal software reminded me recently. It suggested I’d reminisce about it.

I have to say it was a very interesting experience and I like it a lot. I have yet to get more involved in Fedora processes, but I guess that will come after having met more Fedora people at my first FudCon. I’m also slowly getting back into GStreamer development, which works quite well so far. The hardest part is convincing various people inside the company that GStreamer is not “a buggy piece of crap”, which is kinda hard when they show up with weird gst-launch pipelines that expose bugs in 5 or more different elements. On the one hand the flexibility of gst-launch is great, on the other hand it sucks if it doesn’t work once you try something out of the ordinary. Wonder how to solve that…
And I didn’t manage to get the Cairo guys to release Cairo 1.10. If somebody knows how to fix that, please do it.

I also wanted to point out two things that surprised me about my job, one of them very positively, the other not so much.

the great thing

Red Hat engineers are great at working together across communities. If a kernel guy has a problem with GStreamer, they poke me and if I have a problem with building rpms, I can poke its maintainers. I feel that in the “upstream” world, developers often stay in their own community and don’t reach out to others. This leads to ugly workarounds in code and bad blood between people when they blame each other for bugs. I think Red Hat is a model for this and they should go out into the world and give talks about how we achieve this.

the not so great thing

We envy Ubuntu’s fame. I often see people post links to news sites or blogs where Ubuntu gets praised for a new feature that was written by Red Hat people (and of course it was in Fedora so much earlier). And then everybody makes nasty jokes about Ubuntu just stealing all the fame and giving no credit.
It would be a lot nicer if we could be happy that the Ubuntu community likes the software we write. We would be happier and feel better about helping Ubuntu people. Of course, it would be nice if the Ubuntu world would credit us a bit more for our work, but then we probably need to market the stuff we do more.

Can I say this?

Diversity is not a good thing.

Everybody complains when there’s more than one Open Source project for a given purpose – GNOME vs KDE, Swfdec vs Gnash, … And we all agree that the Internet is fine with IP and doesn’t need support for Appletalk or IPX. So why is diversity suddenly a good idea for i18n and l10n?

We don’t need more women in Open Source.

See question above. Also, I’ve never seen any studies on this. Is gender equality something important for progress in Open Source? All the papers I know only point out that there’s too little women. I’ve never seen questioned if that’s maybe a good thing.
After all, we know that proprietary software is worse than Open Source. And more women work on that.

The current ways Open Source attract women are a failure.

There are lots of very vocal groups about women in Open Source. But I’ve not seen them make a difference. There’s still way too little women. How come?

equality

It’s accepted wisdom that all men are created equal. A lot of people derive from this the idea that all people should be treated equally and have the same rights. It’s the basis for democracy after all. As such, many communities strive to achieve equality and even set it as their explicit goal.

However, in recent times there were two cases that made me doubt in equality as a good goal for any community. The first was the discussion about Fedora’s target audience, in particular defining the target as someone who “is likely to collaborate [...] with Fedora” which excludes a whole lot of lazy or uninterested people. The other is Lefty’s surveys where nobody wondered that the surveys assume everyone’s opinion is equally important.

Then there are a lot of places where not having equality is normal and everybody would look at you funny if you were to advocate it. Meritocracy is a very positive world in the open source development communities for example. It’s a known fact that the maintainer decides which patches go in and which don’t. (audio of the talk) So what’s the right way here?

I had a discussion about this with my girlfriend and we found a lot of similar places in the real world where groups wanted to appeal to everyone and ended up being unrecognizable from everyone surrounding them: The Greens for example became just another party without any differences. Apple is in the process of losing it’s style – you’re not special anymore if you have an iPhone or an iPod – everybody has one. And Google does evil these days. All of them saw an increase in “market share” in the process though.

So it seems that in the end it all comes down to this: Is it worth giving up one’s values for more market share?

I personally think everybody who tries to be inclusive is betraying his foundations and original goals. I’m not goint to call it “sells their soul”, because it sounds cheesy, but that’s how it feels to me. So GNOME, please do not give up requesting Freedom. And Fedora, please continue to target the people involved in being bleeding edge. Because the people who are crazy enough to think they can change the world, are the ones who do.

Happy New Year

First of all, I need to excuse to my fellow blog readers for never posting a link to the video hackfest conclusions. The page still looks spot-on, maybe apart from the timeline. I’d add half a year to it.

I’ve also just started my new job in Red Hat‘s desktop team. The job description so far just involves hacking on the same old stuff: world domination. I haven’t heard Flash or browsers mentioned, but I think I’ve heard the words video editing and video conferencing. Exciting times ahead for me!

PS: I would have linked my new work email address, but I’m not going to mention it anywhere. These people already had enough Schadenfreude trying to invent connotations for it.

Video Hackfest day 2

Carl stayed true to his awesomeness from yesterday: He updated the hackfest notes with the things we did today. In particular, it includes “hacking ideas” that we’d like to work on.

I’ve spent a lot of time discussing the ideas of my gst-plugins-cairo design with all people. And I have to say I’m happy to say that the general approach has seen excitement from all sides and there doesn’t seem to be any big issues with it. THe best way to summarize it is probably an event from today: Edward ran a gst-launch pipeline as a benchmark for gst-plugins-cairo and it completed in 0.2 seconds.