Swfdec, GStreamer, Cairo, GNOME, me

(I need to put that somewhere because people ask about it and having a little post to explain it is nice.)

What’s it about?
GTK recently introduced the ability to offload graphics rendering, but it needs rather recent everything to work well for offloading video decoding.

So, what do you need to make sure this works?

First, you of course need a video to test. On a modern desktop computer, you want a 4k 60fps video or better to have something that pushes your CPU to the limits so you know when it doesn’t work. Of course, the recommendation has to be Big Buck Bunny at the highest of qualities – be aware that the most excellent 4000×2250 @ 60fps encoding is 850MB. On my Intel TigerLake, that occasionally drops frames when I play that with software decoding, and I can definitely hear the fan turn on.
When selecting a video file, keep in mind that the format matters.

Second, you need hardware decoding. That is provided by libva and can be queried using the vainfo tool (which comes in the `libva-utils` package in Fedora). If that prints a long list of formats (it’s about 40 for me), you’re good. If it doesn’t, you’ll need to go hunt for the drivers – due to the patent madness surrounding video formats that may be more complicated than you wish. For example, on my Intel laptop on Fedora, I need the intel-media-driver package which is hidden in the nonfree RPMFusion repository.
If you look at the list from vainfo, the format names give some hints – usually VP9 and MPEG2 exist. H264 and HEVC aka H265 are the patent madness, and recent GPUs can sometimes do AV1. The Big Buck Bunny video from above is H264, so if you’re following along, make sure that works.

Now you need a working video player. I’ll be using gtk4-demo (which is in the gtk4-devel-tools package, but you already have that installed of course) and its video player example because I know it works there. A shoutout goes out to livi which was the first non-demo video player to have a release that supports graphics offloading. You need GTK 4.14 and GStreamer 1.24 for this to work. At the time of writing, this is only available in Fedora rawhide, but hopefully Fedora 40 will gain the packages soon.

If you installed new packages above, now is a good time to check if GStreamer picked up all the hardware decoders. gst-inspect-1.0 va will list all the elements with libva support. If it didn’t pick up decoders for all the formats it should have (there should be a vah264dec listed for H264 if you want to decode the video above), then the easiest way to get them is to delete GStreamer’s registry cache in ~/.cache/gstreamer-1.0.

If you want to make sure GStreamer does the right thing, you can run the video player with GST_DEBUG=GST_ELEMENT_FACTORY:4. It will print out debug messages about all the elements it is creating for playback. If that includes a line for an element from the previous list (like `vah264dec` in our example) things are working. If it picks something else (like `avdec_h264` or `openh264dec`) then they are not.

Finally you need a compositor that supports YUV formats. Most compositors do – gnome-shell does since version 45 for example – but checking can’t hurt: If wayland-info (in the wayland-utils package in Fedora) lists the NV12 format, you’re good.

And now everything works.
If you have a 2nd monitor you can marvel at what goes on behind the scenes by running the video player with GDK_DEBUG=dmabuf,offload and GTK will tell you what it does for every frame, and you can see it dynamically switching between offloading or not as you fullscreen (or not), click on the controls (or not) and so on. Or you could have used it previously to see why things didn’t work.
You can also look at the top and gputop variant of your choice and you will see that the video player takes a bit of CPU to drive the video decoding engine and inform the compositor about new frames and the compositor takes a bit of CPU telling the 3D engine to composite things and send them to the monitor. With the video above it’s around 10% on my laptop for the CPU usage each and about 20% GPU usage.

And before anyone starts complaining that this is way too complicated: If you read carefully, all of this should work out of the box in the near future. This post just lists the tools to troubleshoot what went wrong while developing a fast video player.

Found this today while playing around, thought people might enjoy this riddle.

$> echo test.c
typedef int foo;
int main()
{
  foo foo = 1;
  return (foo) +0;
}
$> gcc -Wall -o test test.c && ./test && echo $?

What does this print?

1. 0
2. 1
3. Some compilation warnings, then 0.
4. Some compilation warnings, then 1.
5. It doesn’t compile.

I’ll put an answer in the comments.

An idiom that has shown up in GTK4 development is the idea of immutable objects and builders. The idea behind an immutable object is that you can be sure that it doesn’t change under you, so you don’t need to track changes, you can expose it in your API without having to fear users of the API are gonna change that object under you, you can use it as a key when caching and last but not least you can pass it into multiple threads without requiring synchronization.
Examples of immutable objects in GTK4 are GdkCursor, GdkTexture, GdkContentFormats or GskRenderNode. An example outside of GTK would be GBytes

Sometimes these objects are easy to create using a series of constructors, but oftentimes these objects are more complex. And because those objects are immutable, we can’t just provide setters for the various properties. Instead, we provide builder objects. They are short-lived objects whose only purpose is to manage the construction of an immutable object.
Here’s an example of how that would look:

Sandwich *
make_me_a_sandwich (void)
{
  SandwichBuilder *builder;

  /* create builder */
  builder = sandwich_builder_new ();
  /* setup the object to create */
  sandwich_builder_set_bread (builder, white_bread);
  sandwich_builder_add_ingredient (builder, cheese);
  sandwich_builder_add_ingredient (builder, ham);
  sandwich_builder_set_toasted (builder, TRUE);
  /* free the builder, create the object and return it */
  return sandwich_builder_free_to_sandwich (builder);
}

This approach works well in C, but does not work when trying to make the builder accessible to bindings. Bindings need no explicit memory management, so they want to write code like this:

def make_me_a_sandwich():
    # create builder
    builder = SandwichBuilder ()
    # setup the object to create
    builder.set_bread (white_bread)
    builder.add_ingredient (cheese)
    builder.add_ingredient (ham)
    builder.set_toasted (True)
    # create the object and return it
    return builder.to_sandwich ()

We spent the hackfest arguing about how to create a C API that works for both of those use cases and the consensus so far has been to turn builders into refcounted boxed types that provide both the above APIs – but advertises the C-specific parts only to the C APIs and the binding-specific parts only to bindings. So the C header for the above examples would look like this:

SandwichBuilder *sandwich_builder_new (void);
/* (skip) in bindings */
Sandwich *sandwich_builder_free_to_sandwich (SandwichBuilder *builder) G_GNUC_WARN_UNUSED_RESULT;

/* to be used by bindings only */
GType *sandwich_builder_get_type (void) G_GNUC_CONST;
SandwichBuilder *sandwich_builder_ref (SandwichBuilder *builder);
void sandwich_builder_unref (SandwichBuilder *builder);
Sandwich sandwich_builder_to_sandwich (SandwichBuilder *builder);

/* shared API */
void sandwich_builder_set_bread (SandwichBuilder *builder, Bread *bread);
void sandwich_builder_add_ingredient (SandwichBuilder *builder, Ingredient *ingredient);
void sandwich_builder_set_toasted (SandwichBuilder *builder, gboolean should_toast);

/* and in the .c file: */
G_DEFINE_BOXED_TYPE (SandwichBuilder, sandwich_builder, sandwich_builder_ref, sandwich_builder_unref)

And now I’m off to review all our builder APIs so they conform to this idea.

So it’s the new year and everybody is keenly adding new resolutions which reminded me I wanted to write this down. And because this is long and important to me, I don’t want it to get lost in some social media. So I decided to put it here. It is personal and has nothing to do with GNOME, so feel free to skip it if that’s what you came for.

I decided to start running some time in 2013 and I consider it one of my most impressive achievements of last year. I originally did it to have a release for the anger after my breakup, but it quickly moved into a case of I didn’t know working out would do that to me. Let’s see what happens next. The way I went about it wasn’t very clear to me at the beginning but I reflected on it quite a lot and a bunch of rules became apparent in the process. They worked really well for me, but would probably involve serious challenges for differently structured people (ie. maybe you :))

1. Make sure I run tomorrow
   I decided early on that I wanted to get into a daily routine so I didn’t have an excuse to not run on any day. With twice-a-week routines or similar I always end up with “I don’t have to do it today” and putting it off until forever. But by caring about tomorrow I had to make sure that I was having fun today. If I didn’t enjoy myself or if I overpowered myself I knew I wouldn’t want to go tomorrow. And I’ve learned recently that the most important thing is spending time on it, so that’s what this is about.
2. Don’t set any goals
   I didn’t want to set any measurable goals because both reaching the goal and not reaching it would be problematic. If I did not reach a goal I would be demotivated and think of myself as a loser. It would force me to overcome that feeling. If I did reach the goal, I would get a feeling of accomplishment, of being done. And that might stop me from going out tomorrow. After all, I’m done.
3. Try to improve
   This is as generic as it gets. But it was important. I always wanted to do better than yesterday. Or last week. Or whatever I felt I could improve on. Not slacking, improving.

I started in earnest after the temperature remained above the freezing point, which was in the mid of March. I picked the dumbest route I could find, which was a 1.4km lap around the block. In retrospect that was a good choice because there is absolutely no traffic I need to think about and I can shut off my brain and just run. The other great thing was that by running a lap I can decide after each lap if I want to run another one. As I never knew in advance how well I feel, this has probably made me run quite a distance more than I would have otherwise.

The first few days I was running 200m, walking 300m, and repeat that. Luckily, nobody would see me when I did it at close to freezing temperatures at 2:00am. But luckily, improvements came quickly and after 2-3 weeks I was already doing that for 2 laps and the running distances started to become longer than the walking part. By June (after 3 months), I upped that to 3 laps in sunshine and quickly after that I ran 4 laps (that is 5.6km) for the first time. Though it would take until October for me to make 4 laps my default distance. And while at the beginning my time for 4 laps was around 35min, it’s now at below 31min. I don’t think I cracked the 30min yet, but I don’t measure myself exactly. This goes back to what I said above about not wanting to have anything concrete, as that usually leads to me starting to compare seconds and then being disappointed when I miss some goals. So all I do is count the songs that played on my mp3 player while I was out and then roughly summing up their lengths.

So how did my body like the fact that I was straining it so much? On the one hand, he seemed to be okay with it. I didn’t lose any weight – probably because I’m not overweight, if rather close – but my body form readjusted in a way that made me require new pants. My quarterly Diabetes measurements also improved noticeably. On the other hand, my body didn’t like to be exercised. I got to learn about a lot of leg muscles I didn’t know existed. My knees were not happy the first two months. But because I could stop running after every lap I was able to quickly adjust to growing pains and just run less if muscles were cramping too much or some tendons complained. However, it took until November until I was not having any issues anymore and enjoyed running up escalators two steps at once again.

What helped me a lot was that I had a bunch of friends around me that run themselves and could give me good insights into their own experiences. And even if they still run a 5k faster than I do, I run way more than them so it’s only a question of time until I catch up. What also helped was that I got the right equipment for running and had no issues with blisters on my feet or having clothing that felt light without being cold in winter temperatures. So there never was any outside temptation that I could use as an excuse to be weak and stop running.

So in summary I ran somewhere between 750 and 1000km in 2013. I got myself into a workout where I average 5.6km each day in slightly more than 30 minutes. My dreams for the future are to get that time to below 28min so the average speed goes above 12km/h, to run 10km for the first time and to get rid of the walking I still do in between. But most of all, I wish to enjoy running tomorrow.

So I’ve been to the 29th Chaos Communication congress or as it’s called 29C3. 6000 people over 4 days 10 minutes from my door. Impressive. Short list of highlights from a person that’s used to LCA or GUADEC:

• entirely volunteer-run
• the audience is young
• very wide-ranging topics (security, politics, activism, lockpicking – I only missed engineering)
• international high-caliber varied talks (my favorites being Jacob Appelbaum, DJB, Violet Blue)
• Internet works all the time (8 of 30Gbps max usage)
• own cell phone network
• very central location
• a really nice mood
• great marketing and press cooperation
• working live streams from FEM of every talk
• every talk in German live-translated into English
• videos available on Youtube and via FTP on the same day.
• I didn’t see any numbers, but would expect ~10% females.
• the same gender issues exist, too.
• I very much enjoyed the Germanness of the conference.
• Everyone spoke English fluently. So I probably shouldn’t be proud of my English abilities.

My only problem: I perceive myself as an engineer, not as a hacker. It’s extremely interesting, but not quite the world I get excited about. But I’ll definitely be back next year.

I suppose I can’t just leave my last post standing there as-is. I’ll start by listing a bunch of things I consider facts about the GNOME project. I don’t want to talk about solutions here, I just want to list them, because I don’t think they are common knowledge. People certainly don’t seem to talk about them a lot.

core developers are leaving GNOME development.

The most recent examples are Emmanuele and Vincent. Both cite the need to look for something different, there is no hard feelings.

GNOME is understaffed.

This is hard to explain in a short and concise way. For anecdotal numbers: GTK has 1 person working full-time on it (me). Glib doesn’t even have that. I think evolution is in a similar situation (a complete email client). We can also try Ohloh’s statistics for GNOME (they include 131 packages, including GStreamer and NetworkManager). You’ll see a sharp drop off of committers on the first page already which suggests around 20 full-time developers at most.

GNOME is a Red Hat project.

If you look at the Ohloh statistics again and ignore the 3 people working almost exclusively on GStreamer and the 2 working on translations, you get 10 Red Hat employees and 5 others. (The 2nd page looks like 6 Red Hat employees versus 8 others with 6 translators/documenters.) This gives the GNOME project essentially a bus factor of 1.

GNOME has no goals.

I first noticed this in 2005 when Jeff Waugh gave his 10×10 talk. Back then, the GNOME project had essentially achieved what it set out to do: a working Free desktop environment. Since then, nobody has managed to set new goals for the project. In fact, these days GNOME describes itself as a “community that makes great software”, which is as nondescript as you can get for software development.
The biggest problem with having no goals is that you can’t measure yourself. Nobody can say if GNOME 3 is better or worse than GNOME 2. There is no recognized metric anywhere. This also leads to frustration in lots of places.

GNOME is losing market- and mindshare.

I don’t want to point out Linus’ bashing, but a bunch of very pragmatic facts that all together lead to fewer GNOME users and developers:

• Distros are dropping GNOME for other environments instead of working with GNOME.
• Previous supporters of GNOME are scaling back their involvement or have already dropped GNOME completely.
• Most important desktop applications have not made the switch to GNOME 3. From talking to them, it’s not a priority for most of them.
• The claimed target users for GNOME are leaving desktop computers behind for types of devices GNOME doesn’t work on.

I won’t be attending GUADEC this year. I don’t feel like I would be productive in the current state of things. The only event I will be missing is the keynote of Jacob Appelbaum. That is all for now.

Morten thinks he knows where the application developers are.

But it turns out that when you don’t want apps, you don’t get any app developers either. And why would we want apps? They just break when we redesign our distros every 6 months anyway.

git blame --reverse gtk-2-24...master gtk/gtkwidget.c

Just in case you always wanted to know who’s deleting all your code.

So, because people were arguing in the comments of my last postabout things along the lines of “it’s easy to fix the bug” and that was not the problem I’m even interested in, let me do a new post outlining the problem I’m interested in.

Here’s the case: I have a security system that is supposed to protect me. Now I want to run a program. Not just any program, but something that I consider kind of important. And now the security system says “I’m afraid I can’t do that.” The program says “Please run $CRYPTIC to allow this.” What do I do?

Essentially there’s 3 things I can do:

1. Trust the security system
2. Trust the violator
3. Trust myself

Now let’s dissect the choices. Choice (1) is what I want to do and how I use my computer. But that only works if the system doesn’t break things on purpose.

Choice (2) is a no-brainer. The violating program is the potential attacker that must be stopped. I don’t want to trust it.

Choice (3) is complicated, because it requires that I educate myself and am aware of what all the programs are doing on my computer right this moment. This is the approach that everybody seems to advocate I should do. There is nuanced ways in which people suggest I’d do it, but in the end everybody wants to make me decide.

I’d like to add some more examples where we have to figure out how to do things, so we’re not talking just about SELinux vs applications:

• My browser vs SSL certificates
• My browser vs malware sites
• My email program vs binary attachments
• My distribution vs random binaries on the Internet
• My package manager vs –nogpgcheck
• Fences vs where I want to go
• “Do not every give your SSN to anyone” vs the real world

In all those cases, we all put varying levels of trust in the security system. Of course, those trust relationships can change over time. But generally, nobody takes a security system seriously that requires exceptions to make work normally. And if I can’t take my security system seriously, I might as well disable it, because it’ll save me from babysitting it all the time. And unless SELinux takes an effort to be taken seriously by me, I will continue to do just that.