Summary of GStreamer Conference 2010

Had a really great time organizing and running the GStreamer conference this year. We had a ton of people attending and all the talks where packed (some of them maybe too packed even, like David Schleefs Orc talk. To bad not more people know about Orc because if widely used and deployed it could revolutionize the performance of Linux systems.

For those wondering videos (and slides) of the talk should start appearing online from next week, I will link to them from the conference web page and also send out an announcement.

Was also happy to see LWN picking up on the plans for GStreamer 1.0 (you have to wait a week unless you are a subscriber, but I do recommend to subscribe) that Collabora Multimedias own Wim Taymans announced in his keynote. Together with the rest of the GStreamer community members we will push forward next year to get the long awaited beast out the door, with Collabora Multimedia dedicating Wim to the task. This time it will happen.

In general I also think we managed to put together a good mix of talks at the conference, with a lot of technology talks centered on things like the mobile editing system we are building for GStreamer and MeeGo with Nokia, the advances in the Telepathy VoIP and video conferencing system and the drop inn ready DLNA support through Rygel. It clearly shows how we moved beyond the basic playback/recording scenario in GStreamer and moved on towards more advanced solutions and at the same time how we in the Collabora family are at the heart at driving a lot of those developments. Cheesy as it sounds I am extremely happy about how we are at a point now where we are not only able to provide a fun and challenging work environment for a lot of people, but also have the resources now to drive the development of projects like GStreamer and Telepathy forward beyond just addressing our clients most immediate needs.

I was not the only one enjoying myself at the GStreamer and CE Linux conference, for instance one of the newest members of the Collabora Multimedia family, Mauricio Piacentini and our Qt and KDE multimedia expert was here, reporting on his experience from the conference and talking a bit about our ongoing effort together with Nokia to great some great Qt C++ bindings for GStreamer.

Robert Swain reports on both the fact that his been with us for a year (time flies crazy fast) and on the talk by Benjamin Gaignard about the GStreamer and Android work we have been doing with them over the last year. I saw Harald Welte was a bit disappointed the glue code between GStreamer and Android has not been released, but as Robert points out, there was still major contributions happening from ST Ericsson to the code of GStreamer through this project, fixes and improvements that will greatly help anyone wanting to us GStreamer with Android.

My friend Thomas Vander Stichele also blogged about the event pointing out that we are a decent sized group of GStreamer greybeards at this point, with quite a few developers around who have about 10 years or more of GStreamer experience ;)

Anyway, considering what a great success the conference was I think there is no way forward except doing a new conference next year, so hope to see everyone I met at next years conference. Co-locating with ELCE seemed to work out really well for both us and them, but we have to wait and see what happens next year due to the big changes for the CE Linux conference due to the Linux Foundation merger, maybe Jim Zemlin doesn’t love us for instance :)

And for those I didn’t get to talk to or only speak to for a short while, I apologize but being the first one there was a lot of little things that needed sorting and fixing to make it all run smoothly, so I am sorry if I seemed stressed and distracted :)

GStreamer 2010 Conference starts tomorrow!

So people are starting to pile into the Collabora office here in Cambridge, in preparation for the GStreamer Conference 2010 and the CE Linux Europe 2010. My old friend Jan Schmidt arrived for instance from Australia yesterday and has been putting the finishing touches on his talk today. I strongly recommend seeinghis talk as he is working on something quite nifty. Not to say I don’t recommend the rest of our great list of GStreamer talks.

For those who have arrived early be aware that you can most likely register at the hotel this afternoon already (they where just waiting on some last material to open registration when I was there) and thus save yourself some time tomorrow morning.

Growing the GStreamer family with QtGStreamer

For those who might have missed it we recently starting an effort together with Nokia to create a full set of Qt friendly C++ bindings for GStreamer. This effort is spearheaded by Mauricio Piacentini, building on the original code by and working with George Kiagiadakis, who has also started a blog about the Qt GStreamer bindings. In addition for its use in MeeGo we also hope that this effort will make the KDE community first class citizens of the GStreamer community and thus help push multimedia on the free desktop forward. So be sure to bookmark Mauricios blog so that you can follow his progress, he plans on blogging regularly about the effort. As part of this effort the code will also be hosted along with the rest of the GStreamer modules on freedesktop.org.

GStreamer Conference 2010 just 18 days away!

So there is only 18 days left before GStreamer Conference 2010 opens its doors in Cambridge, UK. So be sure to mark the 26th of October in your calendar. Preparations are going smoothly with final items being sorted at the venue and programs getting printed. Registrations are also going well with around 120 people signed up to attend so far. If you plan on coming but having registered yet be sure to do so quickly, because there is likely to be a last minute rush of registrations and you don’t want to risk not being able to register and attend because the venue capacity was reached.

Also be sure to check out the list of presentations which is packed full of presentations by the GStreamer community best and brightest and leading technology companies.

Also check out the agenda for the CE Linux Europe Conference, which has established itself as the leading European conference for discussing the use of linux in embedded devices. Since it happens in the same venue as the GStreamer conference, on the 27th and 28th, you can very easily combine the two.

Echo cancellation on Linux

So as I blogged about before, Collabora Multimedia has been doing a project with NLnet to improve echo cancellation support under Linux and Pulse Audio. There were and still are a lot of challenges to get it right, but we wanted to try to lay the groundwork for a system wide solution, which is why we decided to try to implement it within Pulse Audio.

For those wondering what echo cancellation actually means, it is a way to resolve the issue that if you record sound from your laptop microphone and at the same time output sound from your speakers, you easily end up with the sound looping, creating an irritating echo effect, which makes doing voice calls on a machine painful and sometimes impossible. Echo cancellation systems basically try to analyse the data coming out of the speakers so that it can filter it out and ignore it when it comes back through the microphone.

The final result is that we have created a virtual device pair which adds echo cancellation, these virtual devices are automatically used by your application if it announces itself as a ‘phone’ application to Pulse Audio. Our Empathy messaging and video conferencing client does this for instance. The bulk of our work is now done and for many use cases things should just work as soon as the output of our effort gets merged into Pulse Audio and packaged by the distributions. There are some open questions left, but we hope that by making this work available and trying to work with people within the ALSA community we will be able to resolve the remaining issues over time. Anyway, let me just give you the small report Wim Taymans wrote to summarize the work and what has been done:

Pulse Audio filter infrastructure

Currently the echo-cancel module is built upon the virtual source and sink examples, which is currently considered to be the Pulse Audio filter infrastructure.

We briefly looked into the ideas for a different generic filter infrastructure for Pulse Audio. Lennart Poetterings first attempts to implement such an infrastructure were put on hold because of the large complexity wrt to latency and rewinds. Because echo-cancellation is not purely a filter (it needs input from both sink and source) we decided to build the echo-cancel module as a virtual source and sink instead.

Echo cancelling for PulseAudio

A new module called ‘module-echo-cancel’ was added to Pulse Audio. The module adds a new echo-cancel source and sink to the existing devices. All samples played to the echo-cancel sink get echo-cancelled from the samples captured from the echo-cancel source.

The module is built so that new echo cancellation algorithms can be plugged in very easily. Two echo cancellers are implemented already, one based on Speex dsp and another based on the code from Andre Adrian.

The echo-cancel source and sink currently proxy the default source and sink in Pulse Audio. This can be changed with the pavucontrol application by changing the source and sink of the virtual streams.

Currently the echo cancellation code can deal with sources and sinks that share a common clock, such as those found on the same sound card.

For devices that don’t share a common clock, we currently don’t have accurate enough timings from pulseaudio yet in order to implement dynamic resampling. Most echo cancelling algorithms are extremely sensitive to drift and fail as soon as resampling is slightly inaccurate. This means that if you record sound using a USB microphone on your webcam for instance, there is a good chance the card will drift and the echo cancellation will cease to work.

Enable echo cancellation in Empathy

Any application can connect to the new echo-cancel source and sink to use the provided echo-cancellation.

The new echo-cancel source and sink are tagged with the ‘phone’ media.role so that the module called ‘module-intended-roles’ will automatically link Empathy to the echo-cancel source and sinks.

Echo Cancellation test application

In the case that echo cancellation doesn’t work on your system there is a good chance it is due to bugs in the audio driver, to make it more easy to test for this and provide useful bug reports we wrote a test application that can be run when echo cancellation doesn’t work. As with all such applications there is of course only a limited range of things we can test for so it will not be able to detect all types of problems, but it should be able to expose some of the more common ones.

Given that the echo canceller has to run on more than just the ALSA API (pulse audio has various backends), we implemented a new Pulse Audio module called ‘module-test’ that will run a set of tests on all existing source and sink devices.

The current tests include measuring the accuracy of the timevalues reported by the internal clock. The accuracy of this clock is one of the most crucial parts in Pulse Audio because it is directly used to estimate when samples will be played or captured. Having an accurate timevalue for when a particular sample was played and recorded is essential to implement echo cancellation.

An application called ‘patest’ is provided that runs the tests and outputs the results on the standard output.

The output of the patest application contains information about the various devices that were tested along with a min/max jitter and drift measured on those devices. The jitter and drift are mostly caused by inaccurate results returned from the ALSA drivers. A normal jitter would be around +-15us, a typical drift would be +-100us.

The module is constructed in such a way that more tests can be added later.

Future plans

We would really like to resolve the issue of using multiple sound cards, ie laptop speakers outputting from the internal sound card and microphone recording using a USB sound card in a webcam. In order to do so will will try to reach out to some of the core ALSA developers and see if we can work together to figure out if something can be done. In some cases the hardware might not be good enough to resolve the problem in software, but we hope that in at last the majority of the cases we will be able to compensate for hardware issues in the driver layer.

Another thing we hope to see is that with this infrastructure in place that maybe some commercial entities decide to open source their echo cancellation algorithms, enabling us to plug in new ones that are more robust in terms of handling clock drifting for instance. Another possibility is that academic projects working on echo cancellation can now find it easy to put their algorithms into a production system easily, so we can see some innovation in this field happen in the open source space.