Category Archives: GStreamer

GStreamer Conference 2014 talks online

For those of you who like me missed this years GStreamer Conference the recorded talks are now available online thanks to Ubicast. Ubicats has been a tremendous partner for GStreamer over the years making sure we have high quality talk recordings online shortly after the conference ends. So be sure to check out this years batch of great GStreamer talks.

Btw, I also done a minor release of Transmageddon today, which mostly includes a couple of bugfixes and a few less deprecated widgets :)

Transmageddon 1.2 ‘All bugs must die’

Update: I had actually managed to disable the VAAPI encoding in 1.2, so I just rolled a 1.3 release which re-enabled it. Apart from that it is identical

So I finally managed to put out a new Transmageddon release today. It is primarily a bugfix release, but considering how many critical bugs I ended up fixing for this release I am actually a bit embarassed about my earlier 1.x releases. There was for instances some stupidity in my code that triggered thread safety issues, which I know hit some of my users quite badly. But there were other things not working properly either, like dropping the video stream from a file. Anyway, I know some people think that filing bugs doesn’t help, but I think I fixed every reported Transmageddon bug with this release (although not every feature request bugzilla item). So if you have issues with Transmageddon 1.2 please let me know and I will try my best to fix them. I do try to keep a policy that it is better to have limited functionality, but what is there is solid as opposed to have a lot of features that are unreliable or outright broken.

That said I couldn’t help myself so there are a few new features in this release. First of all if you have the GStreamer VAAPI plugins installed (and be sure to have the driver too) then the VAAPI GPU encoder will be used for h264 and MPEG2.

Secondly I brought back the so called ‘xvid’ codec (even though xvid isn’t really a separate codec, but a name used to refer to MPEG4 Video codec using the advanced-simple profile.).

So as screenshot blow shows, there is not a lot of UI changes since the last version, just some smaller layout and string fixes, but stability is hopefully greatly improved.
transmageddon-1.2

I am currently looking at a range of things as the next feature for Transmageddon including:

  • Batch transcoding, allowing you to create a series of transcoding jobs upfront instead of doing the transcodes one by one
  • Advanced settings panel, allowing you to choose which encoders to use for a given format, what profiles to use, turn deinterlacing on/off and so on
  • Profile generator, create new device profiles by inspecting existing files
  • Redo the UI to switch away from deprecated widgets

If you have any preference for which I should tackle first feel free to let me know in the comments and I will try to allow
popular will decide what I do first :)

P.S. I would love to have a high contrast icon for Transmageddon (HighContrast App icon guidelines) – So if there is any graphics artists out there willing to create one for me I would be duly greatful

Transmageddon 1.0 released!

It has been a long time in the making, but I have finally cut a new release of the Transmageddon transcoder application. The code inside Transmageddon has seen some major overhaul as I have updated it to take advantage of new GStreamer APIs and features. New features in this release include:

  • Support files with multiple audio streams, allowing you to transcode them to different codecs or drop them from the new file
  • DVD ripping support. So know you can use your movie DVDs as input in Transmageddon, be aware though that you need to install things like lsdvd and the GStreamer dvdread plugin from gst-plugins-ugly for it to become available. And you probably also want libdvdcss installed to be able to transcode most movie DVDs.
  • Another small feature of the release is that you can now set language information on files with one audio stream inside. I hope to extend this to also work with files that have multiple audio streams. If you rip a DVD with multiple audio streams Transmageddon will preserve the existing audio information, so in that case you shouldn’t need to set the language metadata manually.
  • Enabled VP9 support in the code.

There are some other smaller niceties too, like the use of blue default action buttons to match the GNOME 3 style better and I also switched to new icon designed by Jakub Steiner. There is also an appdata file now, which should make Transmageddon available in a nice way inside the new Fedora Software Installer’

Also there is now an Advanced section on the Transmageddon website explaining how you can create custom presets that allow you to do things like resize the video or change the bitrate of the audio.

And last, but not least here is a screenshot of the new version.
transmageddon-1.0-blue-button

You can download the new version from the Transmageddon website, I will update the version in Fedora soon.

Update from GStreamer Hackfest at Google Office in Munich

To give the wider community a chance to see what happened during the GStreamer hackfest last weekend I put together this blog post is based on an summary written by Wim Taymans, so a big thanks to Wim for letting me reuse parts of his summary.)

So last weekend 21 GStreamer hackers got together at the Google office in Munich to spend the weekend hacking on their favourite GStreamer bits. At this point in time we didn’t have any major basic plumbing tasks that needed tackling so the time was spent hacking on a lot of different projects and using the opportunity to discuss design and challenges with each other.

We where 3 people attending from Red Hat and Fedora; Wim Taymans, Alberto Ruiz and myself.

With the Release of GStreamer 1.0 in September 2012, we drastically changed the way memory is handled in the multimedia pipeline and the large body of work is still in exploring, improving and porting elements to this new memory model. We’re also mostly working on improving the existing elements with comparatively little new infrastructure work.

We’re also seeing a lot of people from different companies that contribute significant amounts of code to the official GStreamer repositories. This has traditionally been a much more closed effort with various pieces of code living in multiple repositories, especially for the hardware acceleration bits. It is good to see that the 1.0 series brings all these efforts together again with more coordination and a more coherent story.

HW acceleration

One of the large ongoing tasks is to improve our support for hardware accelerated decoding, effects and display. With 1.0 we can finally get this done cleanly and efficiently in very many use cases.

Matthew Waters to flew in from Australia to move the gst-plugins-gl set of plugins to the core GStreamer plugins packages. He has been working on these plugins for a while now. Their goal is to use OpenGL to apply operations on the video, like rotation on a cube or applying a shader. With the 1.0 memory management it becomes possible to do this efficiently with a minimal amount of texture upload/downloads. More work is needed here, we can optimize things some more by delaying the work and running the shaders as part of the rendering operation.

Andoni Morales (Fluendo) has also been working on improving hardware acceleration on android. He used some of the new features of 1.0 to make the android codecs use zero-copy by implementing the texture-upload meta data on buffers. This allows the video sink to efficiently create a texture from the decoded data for display. Andoni also ported winks, a video capture source on Windows, to GStreamer 1.0.

Nicolas Dufresne (Collabora) has been working on adding a new set of decoders based on the mem2mem API in v4l2. Not many drivers provide this API yet but it is implemented in some Samsung Exynos SOCs. We would also like to support other m2m operations later, such as color conversion but for that we need to make some of our base classes support the required asynchronous behaviour of mem2mem. The memory management in our v4l2 elements has been gone through several iterations of improvements during the 1.0 cycle but it still is not entirely what it should be. We agreed on what we should do to fix this in the near future. We also briefly discussed the need for a new event that can be used to reclaim memory from a pipeline; many elements that use hardware buffers need to free those before they can negotiate a new format with the hardware so we need a way to make that possible.

Mathieu Bourron (Collabora) has been working on libva, the library for GPU based video decoding and encoding on Intel hardware, and spent his time at the Hackfest fixing up the SPU overlay element to enable hardware accelerated subpicture overlays in the video sink. Traditionally GStreamer would use the CPU to overlay the subpictures (of DVD, for example) on top of the video images. With new GL-based sinks, and hardware accelerated decoders this is very undesirable and can be done much more efficient as part of the final rendering. In 1.0 we have the infrastructure to delay this overlay operation by attaching extra metadata (with the subpicture) to the video images when the video sink knows how to overlay them. We have been doing this with subtitles in cluttersink and other sinks for a while now and soon we can also do this with subpictures.

Plugin Hacking

Arun Raghavan, GStreamer hacker and Pulse Audio maintainer, worked on disabling the audio and video filters in playbin when passthrough mode was selected. In passthrough mode, a video or audio sink can directly handle the encoded media (think a bluetooth headset that can handle mp3 directly or a hardware sink that takes encoded data). He expaned on that work in blog entry.

As a cool hack, Arun also made a source element to read from torrent files so you can watch a movie while you torrent it. He provides more information on that element in his blog, it is actually really cool.

Thiago Santos (Collabora) was continuing with his work to improve the DASH demuxer, reworking the buffering code to make it buffer less and smoother. Dash is one of the new formats (with HLS and MSS) to stream media over HTTP while adapting to bandwidth changes. On the server side, it makes media available in various bitrates while a client switches between bitrates depending on its measured network conditions. Andoni Morales also worked on a new dashsink element that implements the server side of the DASH
format.

Mathieu Duponchelle, a former GSoC student was trying to improve support for seeking in MPEG Transport Streams in order to use them in PiTiVi. Seeking in MPEG TS is not an easy thing because they are really optimized for streaming only. He got help from Thibault Saunier (Collabora), who was also hacking on PiTiVi and who was preparing a new release of gnonlin, GES and gst-python 1.2 (which he released on Sunday). Mathieu is one of the developers able to work fulltime on PiTiVi now thanks to the PiTiVi fundraiser, so be sure to contribute to that!

Jan Schmidt (Centricular), a long time GStreamer core hacker was working on debugging some DVB issues and also ended up taking part in a lot of the general design and troubleshooting discussions happening during the hackfest, helping other people move forward with their projects.

Long time GStreamer hacker Edward Hervey (Collabora) was planning to do a lot of DVB hacking but had to give up on that effort when it was clear that Google had signal isolated the office for security reasons, so there was no DVB signal in the Google office. Instead he worked on merging some pending DVB patches and implemented GAP support in the mpeg transport stream plugin. GAP support deals with streams that have long periods of no media (like missing audio for some time in DVD). It makes sure that downstream elements keeps processing the silence instead of waiting for more data.

Applications

Meg Ford, a GSOC student mentored by Sebastian Dröge (Centricular) was working on Gnome Sound Recorder and fixing up the last bugs, preparing it for a new release.

Myself, Christian Schaller (Red Hat) was on a bug fixing spree in Transmageddon (a transcoding application written in python and GStreamer) and managed to reduce the number of known bugs to only 1. Fixed that last bug once I got home, so now I just need to hammer at Transmageddon for a bit to make sure I caught all the corner case issues so I can do a major new release with new features such as handling files with multiple audio streams, handling DVD ripping, handling VP9 encoding, handling setting audio stream language information, reducing decoding overhead for streams that we are going to throw away and more. Also had help reviewing and cleaning up the Transmageddon code from Alberto Ruiz, freeing Transmageddon from some ugly code that had survived many library updates and rewrites.

Alessandro Decina(Spotify) kept working on his patches to update the Firefox GStreamer backend to GStreamer 1.0. We hope to deploy this work in Fedora in the not to distant future. As a hack for the hackfest he provided patches to implement audio and video capture.

Wim Taymans (Red Hat) was hacking on a new library that can parse and generate MIKEY messages (RFC 3830). He want to use this in the GStreamer RTSP server to negotiate SRTP (secure RTP) encryption parameters.

We had 2 people from the Swedish company AXIS, who provide network cameras that all run GStreamer and who contribute on a regular basis to the RTP and RTSP elements and libraries. Ognyan Tonchev was mostly writing some unit tests for RTSP and multicast handling in the RTSP server. Sebastian Rasmussen had been hacking on our watchdog element and the payloaders.

Infrastructure

Long time GStreamer hacker Stefan Sauer (Google) gave a demo of his idea for a tracing infrastructure in GStreamer. The idea is to place trace macros at strategic places that would send structured data to pluggable tracer modules. Some of the tracer modules could, for example measure CPU usage of a plugin or measure the latency. The idea would be to gradually replace our extensive (but unstructured) logging with this new trace infrastructure. This would allow us to do new interesting things, like send debug log to a remote machine or produce STF (Structured Trace Format) to analyse with standard tools. No immediate plans were made to merge this but there seems to be very little resistance to get this merged soon.

Core hacker Sebastian Dröge (Centricular) has been going over the current Stream selection ideas. One of the long outstanding issues is that of switching streams between different languages: you have a movie in different languages and you want to switch between them. To achieve low-latency old data should be kept around for the streams that are not currently selected and be quickly and sent to the audio device. The idea is a combination of events to select a stream and to have the demuxer seek back in the stream on
switches. No final conclusion or plan that can solve all requirements has been reached yet.

Also investigations have begun to make decodebin deal with renegotiations. For example, when a new stream is selected, we might need to use a different decoder for this stream but also when new input is received, decodebin should be able to reconfigure itself. The decodebin code is a complicated beast so any change to it should be done carefully.

GStreamer maintainer Tim-Philipp Müller (Centricular) spent his time merging the new device probing and monitoring API (written by Olivier Crête from Collabora) that had been sitting in bugzilla for a while now. The purpose is to be able to probe devices and their capabilities such as v4l2 and ALSA devices. It’s also possible to be notified when devices appear and disappear in the system. An implementation for pulseaudio devices and another for v4l2 devices using gudev has been committed as well. This reimplements a
feature that was in 0.10, but got cut from 1.0 due to us not being happy with the old design. One of the complications with that was the fact that we ran out of bits in one of our enums so we needed to find a good solution for that.

We briefly discussed how to implement the SKIP seek flag. This extra flag can be used when doing fast forward or reverse and instructs the decoders that it is allowed to throw away data to more efficiently perform the trick mode (at reduced accuracy). There was a prototype for AVI playback that I implemented once that we discussed a bit. We’ll see if someone takes up the task to finalize this work and implement SKIP mode in more demuxers.

I took some photos during the event to capture the spirit and put them on Google Plus for your viewing pleasure.

A big thank you to Google for hosting us and providing us with free lunch and free drinks through the weekend.

PiTiVi fundraiser pass the 10 000 Euro mark

Hi, so I wrote A blog entry asking people to contribute to the PiTiVi fundraiser effort last week. I am happy to see them having already reached 10K Euro, but their goal of course is to get more. I am also happy to say that the GStreamer Project decided to support the fundraiser using some of the money we earned over the years doing Google Summer of Code and organizing the GStreamer Conference, which added 2500€ to the effort. You can read about the GStreamer donation here.

Anyway, I would once again want to ask people to contribute to this effort. There is a proven team behind this fundraiser, so this is not like a kickstarter where people are starting from scratch and you have no idea where they will end up going or what they might achieve. This is an existing project that just needs some polish to get it to critical mass. So visit the fundraising page and make your pledge.

PiTiVi Fundraising Campaign – Why you should donate

So the PiTiVi announced the PiTiVi fundraising campaign on Friday. I sincerely hope they are successful, because I think we really need a good non-linear video editor that runs on the Linux desktop, especially one that is built on top of GStreamer and thus sharing the core multimedia infrastructure of the rest of your desktop. The current PiTiVi team got the right skills and enthusiams in my view to truly pull this off, and their project is scoped in a manner that makes me believe they can pull it off. PiTiVi is already functional and this fundraiser is more about accelerating ongoing development as opposed to creating something from scratch. And their funding requirements for reaching the base milestone is rather modest, for example if just the employees of the 3 main linux distribution companies pitched in 3-4 Euro it would be enough to cover the base funding goal.

But I think this fundraiser is important also beyond the PiTiVi project, because it can serve as a precedent that it is possible to do significant crowdfunding around open source development and thus open the gate for more projects accelerating their development using it. There are a lot of great open source projects out there created on a volunteer basis, which is great and like PiTiVi they will flourish even without crowdfunding, but crowdfunding can be a great way for developers of the most interesting projects to be able to focus solely on their project for some time and thus accelerate its development significantly. So in the case of PiTiVi, I am sure the team will be able to achieve all the goals they have outlined in the funding campaign even if the fundraiser raises no money, but the difference here is if they do it in 1 year or 5 years.

So personally I donated 60 Euro to the PiTiVi fundraiser and I hope everyone reading this blog entry will do the same. Lets give the people developing this and other great open source tools our support and help them make their great software even better. This fundraiser is done by people passionate about open source and their project, because to be fair, no matter if the efforts ends up raising closer to 30 000€ or closer to 100 000 € it is in no way what anyone can call a get rich quick scheme, but rather modest amounts that will let two talented open source developers spend time working fulltime on a project we all want.

And remember whenever a major project using GStreamer gets a boost, it gives all GStreamer projects a boost. For instance in my own pet project, Transmageddon, I am gotten a lot of help over the years from general improvements in GStreamer done due to the involvement from PiTiVi developers, and I have even ended up copying code from PiTiVi itself a few times to quickly and easily solve some challenges in had in Transmageddon.

Hardware encoding in Transmageddon

So thanks to the new GStreamer 1.x VAAPI package for Fedora I was able to do a hardware encode with Transmageddon for the first time today. This has been working in theory for a while, but due to me migrating Transmageddon from GStreamer 0.10.x to GStreamer 1.x at the wrong time compared to the gstreamer-vaapi development timeline I wasn’t able to test it before.

Bellow is the GStreamer pipeline that Transmageddon use now if you have the Intel hardware encoder packages installed. Click on the image for the full image.
libva-pipeline-extract

I am also close to being able to release the new version of Transmageddon now. Most recent bugs found have turned out to be in various GStreamer elements, so I am starting to feel confident about the current pipeline building code in Transmageddon. I think that during the GStreamer Hackfest in Munich next Month I will make the new release with the nice new features such as general support for multiple audio streams, DVD ripping support and language tag setting support.

transmageddon-git-master

GStreamer Conference 2013 – Haggis edition

So we are quickly approaching the time for GStreamer Conference 2013. This year we will be in Edinburgh, Scotland and the conference will be hosted at the Edinburgh International Conference Centre alongside the Embedded Linux Conference Europe and the Automotive Linux Summit.

The GStreamer Conference 2013 Schedule is now live and as you can see there are a lot of great talks this year too, ranging from OpenGL integration, embedded hardware, new codecs and more. As always the GStreamer Conference is the best place to be to discuss the challenges of multimedia.

The conference will be held on the 22nd and 23rd of October so I strongly recommend you get yourself registered. If you want to attend ELCE or the Autotmotive summit make sure to register for those too and extend your stay in Edinburgh to also cover the 24th and 25th of October.

Looking forward to seeing you all there.

And last but not least, a big thank you to this years sponsors of the GStreamer Conference, especially Platinum sponsor Collabora. Also a big thank you to sponsors Fluendo, Google and for the first time this year Cisco.

And as always a big thank you to Ubicast who as always will be recording and publishing the conference for us. Be sure to check out their recordings from earlier years to find out what the GStreamer Conference is about.

Fedora Wayland update

So our team here at Red Hat have been working intensively with our counterparts at Intel to merge and stabilize the patches to enable Wayland support in GNOME and at the same time looking into what further improvements are needed in the stack. Enabling Wayland support means essentially turning the GNOME Shell into a Wayland compositor as we are not going to be using the sample compositor Weston. Porting to Wayland isn’t just about replacing X calls with Wayland calls, in many cases there is also functionality that was in X that will be done as a separate library for use with Wayland or settings that used to be handled by X that now needs to be stored elsewhere. The development work is starting to come together now and tarballs are being released with initial Wayland support or core modules such as Mutter.

The goal we are still pursuing is to have a tech preview ready for Fedora 20. So what do we mean with a tech preview? Well there will be a quite a few things missing or maybe not working as expected. What we hope to have ready is a system where you can have the option of a Wayland session available in GDM, so that instead of logging into your normal X based session you log into one running Wayland instead. Once in that session you should be able to launch and run some applications, but stability is likely to not be great and we don’t know how well XWayland will work by then, so you will also likely having limited mileage with applications that still rely on X. The goal for the tech preview is not to create something an end user is likely to find very useful, rather it is about lowering the barrier for developers and contributors to get involved and start preparing for the Wayland future.

But we do think that it will at least be in a state where developers can easily start playing with it and where the community can help us find issues and bugs, so that we can reach our goal of having a full featured and stable GNOME running on Wayland ready for Fedora 21.

As a sidenote, our top priority is to make sure that the transition from X to Wayland becomes something an end users doesn’t really need to care about. So the final switch to Wayland over X will only happen once we are as sure as we can be that our users will not be negatively affected by the change. So if we default to Wayland or X for Fedora 21 is still an open question, we hope to default to Wayland of course, but staying with X as the default for one more release is not considered unreasonable if it will help us ensure a smooth transition experience for us users. We have of course also not forgotten that many of our users use the binary graphics drivers so we are working on making sure we have an answer ready for that going forward.

You can find details on the status of the tech preview if you go to this website.

GUADEC, Wayland, Transmageddon and more

So GUADEC is kicking off on Thursday here in Brno. The upcoming event is creating quite a bit of excitement here in the office as many members of the Red Hat team here in Brno has been helping out with the organization of the conference this year, being in the hometown of our biggest engineering office in the world. A series of last minute meetings, calls and arriving banners and packages help create a good buzz ahead of the opening of the conference. We have managed to get a bigger contingent of the Red Hat Desktop team this year than usual, including some members of our X.org/Wayland team, our Spice team and our LibreOffice team, so it will be a great opportunity for our global team to meet face to face in addition to meeting the other members of the community.

One of the items I am looking the most forward to during this GUADEC to is to be able to talk our friends at Intel and figure out how we can effectively work together on Wayland. We have put together a team to accelerate Wayland development inside Red Hat, with the goal of getting it ready for deployment in Fedora. There are of course a lot of things that needs to happen in preparation for this, like getting the GNOME Shell ready to work as a Wayland compositor. We will be looking at a long range of items, but among the more important ones is dealing with input devices in Wayland, like handling multiple mice, touch or Wacom tablets and making sure XWayland works so well that our users will not need to realize they are using Wayland instead of X once we transition over.

Anyway, I hope to share more details on our Wayland plans in the coming weeks and Months, so stay tuned.

I have also been trying to find some spare moments during the last few weeks so that I could celebrate GUADEC here in Brno with a new release of Transmageddon. There are basically three new features I am trying to polish up currently. Handling of files with multiple audiostreams is the most important, with DVD ripping support coming in as a close second and finally being able to tag the language of the audio streams as the third one. As you might imagine the first is also a pre-requisite for the second. I am getting there, although there are still some heavy lifting to do in terms of dealing with multiple audio streams inside Transmageddon in the case where the output doesn’t support multiple audio streams, like in the case of the FLV container format or if you are just stripping off the video to make a mp3 file.

Apart from that I think git master is starting to feel like working code again, I just need to test the hell out of it to catch all regressions my code re-organisation has caused. The new features feel small in some ways, but they took an awful lot of re-factoring as they obliterated some assumptions my code made. I am also quite happy to have managed to hide these new features pretty well for people who don’t need them, so if you are just loading a normal file without multiple audio tracks the UI should look and behave almost exactly as it did before. The same is true if you don’t have a DVD player on your system, you will not get any DVD items in the UI. I still need to make the code check for the availability of the libraries it needs for DVD support since they will not be shipping by default with most distros, and thus I will check for them and only make the DVD handling features available in the UI if they are installed.

So looking forward to seeing you here in Brno in the coming days!!