Update from GStreamer Hackfest at Google Office in Munich

To give the wider community a chance to see what happened during the GStreamer hackfest last weekend I put together this blog post is based on an summary written by Wim Taymans, so a big thanks to Wim for letting me reuse parts of his summary.)

So last weekend 21 GStreamer hackers got together at the Google office in Munich to spend the weekend hacking on their favourite GStreamer bits. At this point in time we didn’t have any major basic plumbing tasks that needed tackling so the time was spent hacking on a lot of different projects and using the opportunity to discuss design and challenges with each other.

We where 3 people attending from Red Hat and Fedora; Wim Taymans, Alberto Ruiz and myself.

With the Release of GStreamer 1.0 in September 2012, we drastically changed the way memory is handled in the multimedia pipeline and the large body of work is still in exploring, improving and porting elements to this new memory model. We’re also mostly working on improving the existing elements with comparatively little new infrastructure work.

We’re also seeing a lot of people from different companies that contribute significant amounts of code to the official GStreamer repositories. This has traditionally been a much more closed effort with various pieces of code living in multiple repositories, especially for the hardware acceleration bits. It is good to see that the 1.0 series brings all these efforts together again with more coordination and a more coherent story.

HW acceleration

One of the large ongoing tasks is to improve our support for hardware accelerated decoding, effects and display. With 1.0 we can finally get this done cleanly and efficiently in very many use cases.

Matthew Waters to flew in from Australia to move the gst-plugins-gl set of plugins to the core GStreamer plugins packages. He has been working on these plugins for a while now. Their goal is to use OpenGL to apply operations on the video, like rotation on a cube or applying a shader. With the 1.0 memory management it becomes possible to do this efficiently with a minimal amount of texture upload/downloads. More work is needed here, we can optimize things some more by delaying the work and running the shaders as part of the rendering operation.

Andoni Morales (Fluendo) has also been working on improving hardware acceleration on android. He used some of the new features of 1.0 to make the android codecs use zero-copy by implementing the texture-upload meta data on buffers. This allows the video sink to efficiently create a texture from the decoded data for display. Andoni also ported winks, a video capture source on Windows, to GStreamer 1.0.

Nicolas Dufresne (Collabora) has been working on adding a new set of decoders based on the mem2mem API in v4l2. Not many drivers provide this API yet but it is implemented in some Samsung Exynos SOCs. We would also like to support other m2m operations later, such as color conversion but for that we need to make some of our base classes support the required asynchronous behaviour of mem2mem. The memory management in our v4l2 elements has been gone through several iterations of improvements during the 1.0 cycle but it still is not entirely what it should be. We agreed on what we should do to fix this in the near future. We also briefly discussed the need for a new event that can be used to reclaim memory from a pipeline; many elements that use hardware buffers need to free those before they can negotiate a new format with the hardware so we need a way to make that possible.

Mathieu Bourron (Collabora) has been working on libva, the library for GPU based video decoding and encoding on Intel hardware, and spent his time at the Hackfest fixing up the SPU overlay element to enable hardware accelerated subpicture overlays in the video sink. Traditionally GStreamer would use the CPU to overlay the subpictures (of DVD, for example) on top of the video images. With new GL-based sinks, and hardware accelerated decoders this is very undesirable and can be done much more efficient as part of the final rendering. In 1.0 we have the infrastructure to delay this overlay operation by attaching extra metadata (with the subpicture) to the video images when the video sink knows how to overlay them. We have been doing this with subtitles in cluttersink and other sinks for a while now and soon we can also do this with subpictures.

Plugin Hacking

Arun Raghavan, GStreamer hacker and Pulse Audio maintainer, worked on disabling the audio and video filters in playbin when passthrough mode was selected. In passthrough mode, a video or audio sink can directly handle the encoded media (think a bluetooth headset that can handle mp3 directly or a hardware sink that takes encoded data). He expaned on that work in blog entry.

As a cool hack, Arun also made a source element to read from torrent files so you can watch a movie while you torrent it. He provides more information on that element in his blog, it is actually really cool.

Thiago Santos (Collabora) was continuing with his work to improve the DASH demuxer, reworking the buffering code to make it buffer less and smoother. Dash is one of the new formats (with HLS and MSS) to stream media over HTTP while adapting to bandwidth changes. On the server side, it makes media available in various bitrates while a client switches between bitrates depending on its measured network conditions. Andoni Morales also worked on a new dashsink element that implements the server side of the DASH
format.

Mathieu Duponchelle, a former GSoC student was trying to improve support for seeking in MPEG Transport Streams in order to use them in PiTiVi. Seeking in MPEG TS is not an easy thing because they are really optimized for streaming only. He got help from Thibault Saunier (Collabora), who was also hacking on PiTiVi and who was preparing a new release of gnonlin, GES and gst-python 1.2 (which he released on Sunday). Mathieu is one of the developers able to work fulltime on PiTiVi now thanks to the PiTiVi fundraiser, so be sure to contribute to that!

Jan Schmidt (Centricular), a long time GStreamer core hacker was working on debugging some DVB issues and also ended up taking part in a lot of the general design and troubleshooting discussions happening during the hackfest, helping other people move forward with their projects.

Long time GStreamer hacker Edward Hervey (Collabora) was planning to do a lot of DVB hacking but had to give up on that effort when it was clear that Google had signal isolated the office for security reasons, so there was no DVB signal in the Google office. Instead he worked on merging some pending DVB patches and implemented GAP support in the mpeg transport stream plugin. GAP support deals with streams that have long periods of no media (like missing audio for some time in DVD). It makes sure that downstream elements keeps processing the silence instead of waiting for more data.

Applications

Meg Ford, a GSOC student mentored by Sebastian Dröge (Centricular) was working on Gnome Sound Recorder and fixing up the last bugs, preparing it for a new release.

Myself, Christian Schaller (Red Hat) was on a bug fixing spree in Transmageddon (a transcoding application written in python and GStreamer) and managed to reduce the number of known bugs to only 1. Fixed that last bug once I got home, so now I just need to hammer at Transmageddon for a bit to make sure I caught all the corner case issues so I can do a major new release with new features such as handling files with multiple audio streams, handling DVD ripping, handling VP9 encoding, handling setting audio stream language information, reducing decoding overhead for streams that we are going to throw away and more. Also had help reviewing and cleaning up the Transmageddon code from Alberto Ruiz, freeing Transmageddon from some ugly code that had survived many library updates and rewrites.

Alessandro Decina(Spotify) kept working on his patches to update the Firefox GStreamer backend to GStreamer 1.0. We hope to deploy this work in Fedora in the not to distant future. As a hack for the hackfest he provided patches to implement audio and video capture.

Wim Taymans (Red Hat) was hacking on a new library that can parse and generate MIKEY messages (RFC 3830). He want to use this in the GStreamer RTSP server to negotiate SRTP (secure RTP) encryption parameters.

We had 2 people from the Swedish company AXIS, who provide network cameras that all run GStreamer and who contribute on a regular basis to the RTP and RTSP elements and libraries. Ognyan Tonchev was mostly writing some unit tests for RTSP and multicast handling in the RTSP server. Sebastian Rasmussen had been hacking on our watchdog element and the payloaders.

Infrastructure

Long time GStreamer hacker Stefan Sauer (Google) gave a demo of his idea for a tracing infrastructure in GStreamer. The idea is to place trace macros at strategic places that would send structured data to pluggable tracer modules. Some of the tracer modules could, for example measure CPU usage of a plugin or measure the latency. The idea would be to gradually replace our extensive (but unstructured) logging with this new trace infrastructure. This would allow us to do new interesting things, like send debug log to a remote machine or produce STF (Structured Trace Format) to analyse with standard tools. No immediate plans were made to merge this but there seems to be very little resistance to get this merged soon.

Core hacker Sebastian Dröge (Centricular) has been going over the current Stream selection ideas. One of the long outstanding issues is that of switching streams between different languages: you have a movie in different languages and you want to switch between them. To achieve low-latency old data should be kept around for the streams that are not currently selected and be quickly and sent to the audio device. The idea is a combination of events to select a stream and to have the demuxer seek back in the stream on
switches. No final conclusion or plan that can solve all requirements has been reached yet.

Also investigations have begun to make decodebin deal with renegotiations. For example, when a new stream is selected, we might need to use a different decoder for this stream but also when new input is received, decodebin should be able to reconfigure itself. The decodebin code is a complicated beast so any change to it should be done carefully.

GStreamer maintainer Tim-Philipp Müller (Centricular) spent his time merging the new device probing and monitoring API (written by Olivier Crête from Collabora) that had been sitting in bugzilla for a while now. The purpose is to be able to probe devices and their capabilities such as v4l2 and ALSA devices. It’s also possible to be notified when devices appear and disappear in the system. An implementation for pulseaudio devices and another for v4l2 devices using gudev has been committed as well. This reimplements a
feature that was in 0.10, but got cut from 1.0 due to us not being happy with the old design. One of the complications with that was the fact that we ran out of bits in one of our enums so we needed to find a good solution for that.

We briefly discussed how to implement the SKIP seek flag. This extra flag can be used when doing fast forward or reverse and instructs the decoders that it is allowed to throw away data to more efficiently perform the trick mode (at reduced accuracy). There was a prototype for AVI playback that I implemented once that we discussed a bit. We’ll see if someone takes up the task to finalize this work and implement SKIP mode in more demuxers.

I took some photos during the event to capture the spirit and put them on Google Plus for your viewing pleasure.

A big thank you to Google for hosting us and providing us with free lunch and free drinks through the weekend.

PiTiVi fundraiser pass the 10 000 Euro mark

Hi, so I wrote A blog entry asking people to contribute to the PiTiVi fundraiser effort last week. I am happy to see them having already reached 10K Euro, but their goal of course is to get more. I am also happy to say that the GStreamer Project decided to support the fundraiser using some of the money we earned over the years doing Google Summer of Code and organizing the GStreamer Conference, which added 2500€ to the effort. You can read about the GStreamer donation here.

Anyway, I would once again want to ask people to contribute to this effort. There is a proven team behind this fundraiser, so this is not like a kickstarter where people are starting from scratch and you have no idea where they will end up going or what they might achieve. This is an existing project that just needs some polish to get it to critical mass. So visit the fundraising page and make your pledge.

PiTiVi Fundraising Campaign – Why you should donate

So the PiTiVi announced the PiTiVi fundraising campaign on Friday. I sincerely hope they are successful, because I think we really need a good non-linear video editor that runs on the Linux desktop, especially one that is built on top of GStreamer and thus sharing the core multimedia infrastructure of the rest of your desktop. The current PiTiVi team got the right skills and enthusiams in my view to truly pull this off, and their project is scoped in a manner that makes me believe they can pull it off. PiTiVi is already functional and this fundraiser is more about accelerating ongoing development as opposed to creating something from scratch. And their funding requirements for reaching the base milestone is rather modest, for example if just the employees of the 3 main linux distribution companies pitched in 3-4 Euro it would be enough to cover the base funding goal.

But I think this fundraiser is important also beyond the PiTiVi project, because it can serve as a precedent that it is possible to do significant crowdfunding around open source development and thus open the gate for more projects accelerating their development using it. There are a lot of great open source projects out there created on a volunteer basis, which is great and like PiTiVi they will flourish even without crowdfunding, but crowdfunding can be a great way for developers of the most interesting projects to be able to focus solely on their project for some time and thus accelerate its development significantly. So in the case of PiTiVi, I am sure the team will be able to achieve all the goals they have outlined in the funding campaign even if the fundraiser raises no money, but the difference here is if they do it in 1 year or 5 years.

So personally I donated 60 Euro to the PiTiVi fundraiser and I hope everyone reading this blog entry will do the same. Lets give the people developing this and other great open source tools our support and help them make their great software even better. This fundraiser is done by people passionate about open source and their project, because to be fair, no matter if the efforts ends up raising closer to 30 000€ or closer to 100 000 € it is in no way what anyone can call a get rich quick scheme, but rather modest amounts that will let two talented open source developers spend time working fulltime on a project we all want.

And remember whenever a major project using GStreamer gets a boost, it gives all GStreamer projects a boost. For instance in my own pet project, Transmageddon, I am gotten a lot of help over the years from general improvements in GStreamer done due to the involvement from PiTiVi developers, and I have even ended up copying code from PiTiVi itself a few times to quickly and easily solve some challenges in had in Transmageddon.

Hardware encoding in Transmageddon

So thanks to the new GStreamer 1.x VAAPI package for Fedora I was able to do a hardware encode with Transmageddon for the first time today. This has been working in theory for a while, but due to me migrating Transmageddon from GStreamer 0.10.x to GStreamer 1.x at the wrong time compared to the gstreamer-vaapi development timeline I wasn’t able to test it before.

Bellow is the GStreamer pipeline that Transmageddon use now if you have the Intel hardware encoder packages installed. Click on the image for the full image.
libva-pipeline-extract

I am also close to being able to release the new version of Transmageddon now. Most recent bugs found have turned out to be in various GStreamer elements, so I am starting to feel confident about the current pipeline building code in Transmageddon. I think that during the GStreamer Hackfest in Munich next Month I will make the new release with the nice new features such as general support for multiple audio streams, DVD ripping support and language tag setting support.

transmageddon-git-master

Excited about Cockpit

So we had the DevConf conference here in Brno this weekend. One of the projects I am really excited about is Cockpit. Cockpit is a new server administration tool developed by Red Hat engineers which aims at providing a modern looking and userfriendly interface for your servers. There has been many such efforts over the years, but what I feel makes this one special is that it got graphical designers and interface designers involved, to ensure that the user experience is kept in focus instead of being taken hostage by underlaying APIs or systems. Too many such interfaces, be they web based or not tend to both feel and look clunky, for instance sometimes exposing features not because anyone realistically ever would want them, but because the underlying library happen to have a call for it.

Cockpit should also hopefully put the final nail in the coffin for the so called ‘server desktop’. The idea that you need to be able to run a graphical shell using X on your server adds a lot of pain with little gain in my opinion. The Fedora Server product should hopefully become a great showpiece for how nice a Linux server can be to use and configure when you have something like Cockpit available.

There was some nice videos showing what is already in Cockpit shown at the conference so hopefully they will be available online soon. In the meantime I recommend taking a look at the Cockpit web page.

Getting Wayland Input handling ready

I noticed an article on Phoronix today about libinput which made me think I should post a little Wayland update again. So Libinput is developed by Peter Hutterer who is part of the Graphics team here at Red Hat, and our resident input expert. He is developing libinput as part of our work to get Fedora Wayland ready.

That said input is a complex area and if we do end up not having a Wayland option with feature parity with the X.org option in Fedora 21, then not having gotten input sorted in time is the most likely scenario. That said we are still charging ahead with the goal of getting things ready for Fedora 21, but in our last status meeting we did flag input as our biggest risk. Luckily Peter is not alone in his efforts on libinput, there are a healthy amount of community contributions and at Red Hat we have recently had Hans de Goede join Peter on input. So we are doing our utmost to make sure the pieces fall into place.

Our minimum todo list for input that will enable Wayland to be on parity with X for at least 90% of our users is as follows:

  • keyboard works as-is
  • mouse supports left/right-handed button mapping
  • mouse/touchpad middle mouse button emulation
  • touchpad scrolling and tapping
  • touchpad software-emulated buttons work on clickpads
  • touchpad disable-while-typing

But there are of course other items on here too, like Wacom tablet support, which is of interest to a much smaller segment of our users, but still important to get done. We might have to push some of these more niche items onto the F22 timescale.

Also if anyone is wondering about game pads and such, we don’t have any concrete plans around them currently in the context of Wayland, as when we spoke to Valve and the SDL team they currently access game controllers directly through the kernel interfaces, and preferred to keep doing that. So we decided not to try to second guess them on this considering they been doing game development for years and we haven’t :)

Integrating Chrome Applications into your desktop

With the growing popularity of ChromeOS and Chrome applications we have been doing a little research project inside Red Hat to make such applications a bit more integrated into your Fedora desktop. As you might know if you go into the ‘Tools’ menu in Chrome/Chromium there is an option called ‘Create Applications Shortcut’. If you choose that you can turn any web page or chrome application into something easily reached directly from the desktop, and especially with a lot of ChromeApps now working offline this is a quite nice feature. But there are some issues with this setup. First of all it uses the appicon as the application icon, which looks really ugly compared to the other icons on your desktop, secondly it is a little cumbersome to have to go into that menu to set up your application and lastly there is no way of uninstalling it again save from manually deleting the generated .desktop file.

Well our resident Webkit developer, Tomas Popela, has created a Chrome/Chromium extension which you can download from using this link.
To install it you need to go to the extensions page (chrome://extensions/) and enable ‘developer mode’. Once you have done that you can for instance drag and drop the created extension onto the Chrome extension page to install it. Once it is installed it will automatically create a desktop entry for any application you install from the Chrome store, using a nice looking icon. It will also remove the entry again once you uninstall the application.

Some screenshots of this feature in in action.

As you can see the ChromeApp is using its own image in the Shell activity menu and is session managed separately from a 'normal' Chrome window.
As you can see the ChromeApp is using its own image in the Shell activity menu and is session managed separately from a ‘normal’ Chrome window.

angrybird-activity
And you can search and find your Chrome Apps in the GNOME Shell activity overview just like any other application.

Unfortunately the shelf life of this extension is limited as its relying on Chrome supporting npapi, which it will stop doing in April according to current plans. But we are trying to work with Google to see if we can make this standard functionality going forward.

For those interested you can find the source code here on github.

Enjoy!

A thank you to Google from Desktop Linux

We sometimes grumble a bit about Google in the community when they do things we feel are not generally helpful to the overall community. But I think we should be equally good at saying thanks when Google do great things. So thanks to our LibreOffice superstar Caolán McNamara I was made aware that Google has released two new open fonts along with Chrome. So what is so exciting about a new font you say?

Well one of them, called Carlito, is metrically compatible with the current MS default font called Calibri. You can get the font here. It is licensed under the OFL 1.1.

So for those wondering what metrically compatible means, I for sure did when I first heard the term, it basically mean that while the individual glyphs in the font doesn’t look like the Calibri font (that would not be legal), each individual letter has the same height and width as their Calibri counterpart. This means that if you import a document using Calibri into LibreOffice and you don’t have Calibri or a metrically compatible font installed, your document layout would change as the font LibreOffice would need to use instead have letters that might in general be slightly wider for instance. So with Carlito installed this will no longer be a problem, the glyphs might look a bit different, but you can be sure that the overall layout stays the same.

And for certain professions that can be crucial, for instance try speaking to the legal team of your company about them using LibreOffice and they are likely to tell you that they will only do that if they can feel certain that when another lawyer sends them a contract, the layout will not change when they view it, as such changes could at least potentially be the cause of a dispute over the meaning of a paragraph. (That worry was probably the main reason the legal profession stayed with Word Perfect for such a long time, when the rest of the world had moved on.)

So we are now going to get this new font packaged for Fedora and Red Hat Enterprise Linux so soon as possible, to make your productivity experience even better :)

So thank you Google, this is much appreciated!

GStreamer Conference 2013 – Haggis edition

So we are quickly approaching the time for GStreamer Conference 2013. This year we will be in Edinburgh, Scotland and the conference will be hosted at the Edinburgh International Conference Centre alongside the Embedded Linux Conference Europe and the Automotive Linux Summit.

The GStreamer Conference 2013 Schedule is now live and as you can see there are a lot of great talks this year too, ranging from OpenGL integration, embedded hardware, new codecs and more. As always the GStreamer Conference is the best place to be to discuss the challenges of multimedia.

The conference will be held on the 22nd and 23rd of October so I strongly recommend you get yourself registered. If you want to attend ELCE or the Autotmotive summit make sure to register for those too and extend your stay in Edinburgh to also cover the 24th and 25th of October.

Looking forward to seeing you all there.

And last but not least, a big thank you to this years sponsors of the GStreamer Conference, especially Platinum sponsor Collabora. Also a big thank you to sponsors Fluendo, Google and for the first time this year Cisco.

And as always a big thank you to Ubicast who as always will be recording and publishing the conference for us. Be sure to check out their recordings from earlier years to find out what the GStreamer Conference is about.

More movements around Wayland

Thought I post another update with some more news on what we are doing here at Red Hat in preparation for our continued push forward for Wayland on Fedora. Peter Hutterer posted his first draft of the Wayland protocol extension to handle Wacom tablets, you can find more details in his Google Plus post. This effort is also a good illustration of our approach to the Wayland switchover, it is not just about working on some high level bits to port them over to existing APIS or protocols provided by Kristian Høgsberg and the other core Wayland devs, but actively contributing to the whole stack where needed.

Giovani Campagna continued his great work on porting the needed GNOME stack over, with his patches to Clutter now being available in a release.

Our biggest challenge currently is with XWayland, due to some internal projects that had to get priority we have not had the amount of time we hoped to worked on it yet, but we hope to rally a bit in order to get it to a state in F20 where we can at least have our GDM Wayland session working.

I would also like to point out that I seen some headlines around the web where people are excited about the Wayland work happening in Fedora and around GNOME 3.10. And it is exciting as things are falling into place very fast these days for Wayland, but still I ask people to keep their cool a bit as we are branding it a tech preview in F20 for a reason. So for developers we hope that this tech preview will provide an easy to use harness for running Wayland and testing your applications against Wayland, but in no way is this tech preview going to be something that an end user would even want to run. So when I see Slashdot headlines like “GNOME 3.10 Is Now Properly Supported On Wayland” I get a little worried because to me such a headline implies that things are in a much more mature state than they actually are. Our expectation and hope is to have something that can be considered fully working and ready for end users in the Fedora 21 and GNOME 3.12 timeframe, and that is the releases in which I feel such headlines are likely to be warranted.

So yes, the Wayland stuff we are working is cool and I think we have some killer features planned to be deployed on top of it, but lets do this right and not jump the gun here.

GStreamer news and more

Bad Behavior has blocked 703 access attempts in the last 7 days.