Swfdec, GStreamer, Cairo, GNOME, me

Dear Google, why do you think I’m too stupid to watch movies and need a GoogleVideoPlayer.exe? You seem to be able to hand me the file just fine when running on Linux.

Or have you guys recently realized that Linux users are smarter?

Lots of people keep asking this, so here is the authoritative answer to the question “What’s been happening to Swfdec in the last months?”

The bad news is that there haven’t been many new features. At least nothing that would be very visible. The good news is that this is about to change soon. I’ve been spending most of my time understanding the inner workings of Flash’s script engine and reimplementing it in a brand-new script engine for Swfdec. I’ve thrown away about 70% of the glue code to make objects scriptable. Making scriptable objects is now extremely easy. This should speed up coding immensely, both for me and for new contributors.

The biggest issue so far when making Flash files work has been finding subtle bugs. Swfdec is very good at complaining about missing features to stderr (see my last blog), but other things are more subtle. For example: is NaN == NaN? Sure, if you tested it, you know the answer, but if you just coded it, you’ll probably get it wrong. And finding such a problem inside huge 16.000 lines of code (that’s the size of the Google Video Flash plugin) is very hard. So I’ve been looking for ways to find these issues earlier. I hope it works out.

I’ve also developed a habit of writing a test when I’m not sure about a feature, which has helped a great deal. The testsuite has increased from 90 tests to almost 200 tests and there’s another 150-200 tests lying around waiting to be cleaned up and committed.

So, in short, for the user, nothing new to show. For developers, lots of exciting new stuff has landed. If you are a developer, today is a good day to start. Grab the code, head into #swfdec and start hacking. The hard work behind the scenes is done, now it’s time for features.

So someone recently asked me how to hack on Swfdec. And since that might be interesting to others, I thought I’d blog about it. One thing to note: All the action these days is in the “as” branch, which will become Swfdec 0.5 in the near future.

So the first thing I do is download the file I want to make work. Surprisingly, the Swfdec Mozilla plugin allows downloading Flash files from its properties dialog, which you can get to by right-clicking on the playing file.

Now I play the file with the little application (in Swfdec’s git checkout) player/swfplay inside a terminal. It will spew out debugging information. A lot of this debugging information will be about missing features. So I start at the top and implement all those missing features, most of the time by writing accompanying tests. (How to easily write good tests is a post as big as this, so I’ll leave that out for now. Poke me to write it if you care.)

Here’s a few of the more common messages and what to do about it:

SWFDEC: ERROR: swfdec_script.c(198): validate_action: no function for  36 0x24 CloneSprite in v6

This means a bytecode isn’t implemented yet. The bytecode interpreter is in libswfdec/swfdec_as_interpret.c – just add the function there that does the right thing.

SWFDEC: ERROR: swfdec_as_interpret.c(678): swfdec_action_call_method: no function named createEmptyMovieClip

An Actionscript function isn’t implemented. First I figure out what object it belongs to and what it does by looking at Brajeshwar’s excellent Actionscript reference. Then I implement it. Note that a lot of function have really weird effects if called with unexpected parameters. So it really helps to write tests here.

SWFDEC: WARN : swfdec_as_interpret.c(1316): swfdec_action_new_object: Sound is not a constructor

It looks like an object isn’t implemented at all. So I do the same thing as above. I implement it, then run player/swfplay to see which functions on the object are used and implement those.

If I managed to implement all the functions that needed to be implemented and it still doesn’t work, then the hard work starts. That’s where I launch player/swfdebug and step through the code looking if what it does is sane. Swfdebug is a nice little tools for debugging Flash files that unfortunately is a little crashy and accumulates features as I need them. There’s also some tools in the test/ directory that help understanding particular Flash files, in particular test/dump that dumps information about the static content of a file (run with -v for maximum output) or test/swfdec-extract that can extract contents of Flash files to png or svg and I think it even extracts audio these days.

And when I’m finally done doing all this, I have one more file to play with. And most of the time I also know 3 more places that need serious refactoring…

So, if you now want to hack on Swfdec, a good idea is to come to IRC on Freenode, channel #swfdec and start asking questions. If you don’t want to use IRC, you can also mail the mailing list, just reading commit messages there is boring anyway.

Zack has a long blog about browser plugins. And he advocates going the XEmbed route. Originally I wrote this as a comment, but I think more people should know about the mess that is called browser plugins.

Swfdec used to be an out-of-process plugin. That plugin had all the nice features he mentions, but utterly fails for other things:

• translucency. If you use an X window for embedding as in XEmbed, you suddenly need Composite support to be able to do it.
• z-order. It’s absolutely fine to pop up a DIV on top of any plugin. If you go the X-Window route, you’re in for quite a ride since suddenly you have to create an X-Window for (at least) every DIV.
• Input events. You know when you use the mouse wheel on top of an embedded Flash animation and it doesn’t scroll? Or when you press Ctrl-T to open a new tab and none opens because the plugin has keyboard focus? Those.

So how would I solve the current mess? I would go the same route every other application with plugins goes. I’d write one plugin for every application. There’s really no need to have shared plugins. Noone wants to run Rhythmbox plugins in Amarok or Konq plugins in Nautilus.
If every browser had its own plugin system, suddenly Swfdec could use all the nice APIs of the browser. It could integrate its properties with the Konq properties dialog. It could share common properties (“run animations [x] always [ ] once [ ] never”, “sound [ ] enabled [x] disabled”), it could react to events (like ESC stopping animations). And there’ll probably be lots more I’m missing.

I can imagine two reasons why browsers didn’t go that route. The first is closed source plugins. Adobe ships Flash only for Moz plugins. So everyone tries to add support for that one plugin. And the second reason is that there really isn’t a huge demand for plugins. There’s really only Flash and Video players. Integrate those two natively and you don’t even need plugin support at all.

Oh, and one final note. Please stop bringing up the “misbehaving plugins” argument. It’s the reason for so much braindead code trying to cope with “misbehaving” stuff. If some code is broken, fix the broken code, don’t try to work around it. It feels like advocating to render every tab out-of-process, because then one page crashing doesn’t kill the browser…

Everybody knows about multiple solutions to the same problem. Everybody and their mom complains about them and that they should merge or at least work together. Surprisingly they do. At least most of the time. And of course there’s an exception. It’s called distributions. Noone complains about the fact that there’s a huge amount of them. And noone complains about the fact that they don’t work together. And I’m experiencing this on two different fronts now.

Imagine a project where upstream is in some “broken” state. Examples include, but are probably not limited to FFmpeg not doing releases and Mozilla not being able to ship one development platform. So what do the distros do to solve that problem? Right, they get creative. FFmpeg version numbers range from 0.49 to 0d.51. I bet you can guess which one is more recent. Now sometimes upstream gets in a good mood and tries to fix it. (I’ve heard that m4 macro is not always working. ;))

Dear distros, could you make life of us hackers simpler, and when you encounter a broken upstream, get together and work out a solution that makes life easy for us? It would make life a lot easier for us developers of not-broken packages and people that compile from sources like jhbuild.

Dear everyone else, in case you encounter a package that I maintain and that package doesn’t build on your distro of choice, there’s a pretty good chance of a broken upstream and distros being creative. I’ll most likely have a rant in configure.ac in that case. And then: Blame your distro.

PS: Thanks to Olav Vitters my blog moved from Advogato to GNOME blogs. So you can write comments now.

If anyone is interested and able to speak German there’s an
interview about
Swfdec I recently did. I think it turned out quite nice.

<ds> dude. zzuf rocks

Anybody coding an app that has to handle untrusted
input should have used zzuf. A lot.

Miguel,
thanks for pimping
Swfdec. But I think your conclusions are wrong. Flash is
not even close to what I’d suggest as an option for “rich
media”.

For a start, the currently supported Video formats for Flash
are Sorensen Squeeze and On2 VP6. As far as I know, both of
those companies are actively fighting Free implementations
of the codec. That makes Flash video as bad or worse than
VC-1. You can play all 3 of them with FFmpeg btw.
Then there’s the fact that the interaction between Flash and
browsers is crap. You can execute Javascript from the Flash
file and you can execute predefined Flash functions from
Javascript, but there’s no DOM or anything. But what about
integrating Adblock, the “animate only once” option, popup
blocking or all the other things you want integrated?
As for the things you mentioned: The Flash spec is
semi-open, in the sense that you may not use it to write an
interpreter for Flash. That’s equivalent to closed for
everything you care about. And the spec covers the file
format only, not the video formats used or what script
objects are supported.
And you can populate Flash files using remote data. You can
even chose between XML and Flash. The XML needs to be
interpreted, but I’m pretty sure there’s convenience
libraries for you that do that.

And it’s not like the Free alternatives are far behind in
the game. HTML5
has a video tag and SVG is far superior to anything
Flash offers in drawing options. Especially because it’s
scriptable while Flash shapes are not.

There’s 2 things missing:

• Simple animation support: Flash makes creating
  animations extremely easy. Doing the same thing in
  anything Free (HTML, SVG, even Gtk) requires being a good
  code monkey. The example above required NO coding.

• Authoring tools for rich web applications. Depending on
  who you ask, the best Free authoring tool is either vim or
  emacs. There’s the Gimp and Inkscape of course, but neither
  does animations nor scripting, so it’s pretty useless for
  doing rich web apps.

So I can tell you what you want. You want to create an
awesome authoring tool that integrates HTML5 and SVG and
make Webkit and Mozilla handle those. And after that I bet
rich web apps are Free.

Talking about security is “funny”. Like looking how
people react to security issues or talking with other
developers in #gnome-hackers, where I recently claimed that
GNOME’s security sucks and got quite a backlash. Since
writing multimedia players like GStreamer or Swfdec made me
think about security (I’m still far from expert
unfortunately), I’d like to elaborate a bit on that.

Let me first mention one important thing: I don’t just
consider crashes to be stuff that shouldn’t happen.
Everything that disrupts the user’s computing experience
should not happen. Taking too many CPU ressources or too
much memory for too long effects other applications
negatively, so it should be avoided.

The biggest problem for me has always been the question of
trust. When interpreting data I have to ask myself if I
trust that data or not. If I have a number that says
“allocate this much memory”, I better trust it to report a
correct size or I’ll get an out-of-memory problem. A worse
problem are pointers: read (or write to) the memory pointed
to by this address. If you can’t trust a pointer you’ll
likely get a segfault. So you better trust every pointer in
your app. Note that this includes modifying pointers (like
array indexing), which should only be done with trusted
values. Stuff like that quickly gets my head spinning about
which values are trusted and which aren’t. Another example:
Do we trust this string to be valid UTF-8?

A lot of data is read from files these days. Now, what files
can application developers trust? Apparently we trust all
the data a user doesn’t have write access to, since I can
make apps crash easily by removing Glade files, icons or
linked-in libraries. But should applications trust stuff in
the user’s home directory?

Currently a lot of applications trust stuff in the user’s
directories unconditionally. Bash and X execute files in
there for example. However epiphany allows you to save files
from the web in there, too. What do you think clueless users
would do when a web site tells them to download a file named
.xinitrc? Probably nothing, since .xinitrc must be
executable in order to get executed, but we’re already very
close to running “rm -rf ~” on login here. And of course we
suddenly have a lot of untrusted files in the user’s home
directory. Nautilus for example knows this, since the
thumbnailing process has brought down Nautilus quite often
in the past. But I bet a lot of other applications don’t
think about that.

And then there’s the fact that it’s apparently dead
easy to find security holes in Free Software while Windows
products have been exploited by those issues in the
past.
So I guess when the Linux desktop gets interesting for
crackers, we’ll be in for quite a ride. And by “we”, I’m
talking about every application that reads files.

So I’ve managed to write software that a lot of people are
suddenly interested in. It’s an interesting thing to say the
least. However “becoming famous” was nothing like what I’d
had imagined. Here’s what happened after my initial
blogpost:

1. The media picks it up.
   The media is interested in new stories. So they post them.
   However, they often prefer to give those stories their own spin.
   (The really weird stories have luckily already disappeared
   from Google, so no links to them.) None of those journalists
   contacted me about it, they just wrote something.

2. The people pick it up.
   Lots of people start talking about it. How do I know? I
   googled. Only 5 people talked to me about it. Mostly they
   just went into obscure forums or chats to discuss it and
   complain about missing features. Almost nobody of them
   realizes that it’s a really good idea to come to me and ask
   questions. It’s highly motivating if someone shows interest
   in my product. Even if it’s just complaining that it doesn’t
   build on his box. And if you ask, there’s a high chance I
   get interested in solving your problem, especially if it’s a
   pretty simple problem.
   (Side note: The Ubuntu forums are
   lucky to have such a high Google ranking, so I could at
   least find their questions.)

3. When there’s a release, the distros pick it up.
   Distros are really fast at taking your source code and
   sticking it in the unstable branch of their package
   repositories. Lots of them, in particular the big ones,
   contact you to tell you about it and ask questions. The
   process of integrating new software into distros seems to
   work really well. Congrats from me to the distro people for
   that. Great work you’re doing there. In particular Gentoo.

4. People start blaming
   They seem to read everything I have said in the past and
   spin it in their direction. I’m suddenly personally
   responsible for not working on Gnash and fragmenting the
   Free Flash movement, I sure as hell have to provide
   OpenBSD support or my software is crap, stuff like that.
   Again, noone comes and asks, that would be far too
   complicated. And probably wouldn’t be compatible to their
   predetermined opinion.

5. Hackers don’t show up
   I was expecting some interested people to show up and have a
   look at the code. In particular because the code is
   documented far better than all my previous code (including
   GStreamer ;)). I think there have been 2 people so far that
   looked at it. But that’s about it. Probably everyone is busy
   doing their own project. Or people still think doing Flash
   is hard (hint: it’s not).

But then, the few people that have shown up and talked to me
have made a lot of things happen and I guess I’ll cut a new
release soon to make those people happy. In particular:

• People wanted an easy API to embed Flash files. So I
  created libswfdec-gtk. You should be able to play Youtube
  videos in your Gtk app with 20 lines of C.

• Or you could do it in 10 lines of Python with the new Pyswfdec Python
  bindings that Gian Mario
  Tagliaretti has been working on.

• Last but not least I’ve managed to use GStreamer instead
  of ffmpeg/mad for video and audio decoding. While that
  doesn’t sound to exciting at first, it means you can build
  swfdec now without linking to patented multimedia formats
  and still be able to enjoy Youtube. I know at least Fedora
  was very interested in that.

No new work on Flash features so far, since I’ve been
working on a branch redoing the ActionScript interpreter.
Once that hits (that’ll definitely be after next release),
it’ll make a huge number of Flash files supported. But
that’s still some work.

What I’ve been wondering and like to get some feedback from
people active in this area is if it’s a good idea to export
the script interpreter. ActionScript is
almost the same as Javascript (same syntax), so it offers a
sandboxed execution environment. In this case completely
GObject’ified. So you could use it if you wanted to script
Flash files, or you could even add hooks to it in your own
app and download interesting scripts from the web. Think Get Hot new Stuff
with scripts. To me that’s somewhere between total crack and
really exciting.