Category Archives: General

Tutorial for Python, GStreamer and GTK 3

I have wanted to write about programming with GStreamer and Python for a while. Jono Bacon wrote a nice introduction to GStreamer and Python a long time ago, but I want to share with you some specific tips.

At Collabora we work a lot with GStreamer including helping train developers at our customers to be better at GStreamer development. Being the lowly marketing guy at the company I don’t have the programming chops to teach the hard stuff, but I figured I should be able to put together a very simple article which explains some basics and shows of a little GStreamer development trick I have used to great success in Transmageddon.

Part of what triggered getting this little tutorial done was that I am looking into porting Transmageddon to GTK3 after its next release of Transmageddon. To understand how to write a GTK 3 Python application, using the introspection bindings, I decided a good learning tool for myself would be to try to port the 0.0.1 version of Transmageddon. This version was never released, in fact it was me trying to figure out the very basics of programming with GTK+ and GStreamer in Python.

The application litterally consists of a GTK+ UI with two buttons. One is a ‘transcode’ button which when pressed starts a GStreamer transcoding pipeline. The other is my little secret trick, called ‘Debug’. It will when pressed generate a png of the pipeline being run, or not being run for that matter. It has helped me solve a ton of bugs and issues in Transmageddon since I started the project and hopefully it can be a useful trick for you too.

You can find a tarball here with the code below, the .ui file from Glade and a which.py file (which.py is a python version of the Unix which tool, which I found online).

First let me give you the code of the application, I tried to annotate the code in detail to make it easy to follow, even if you haven’t played with either GTK3 or GStreamer before.

 
#!/usr/bin/env python
# Simple example GTK3 + GStreamer 0.10.x Application for transcoding 
# GTK3 using gobject introspection for bindings, GStreamer using manual bindings 
# Also includes how to set up dotfile generation

import sys
import os
import which

# Setting GST_DEBUG_DUMP_DOT_DIR environment variable enables us to have a dotfile generated
os.environ["GST_DEBUG_DUMP_DOT_DIR"] = "/tmp"
os.putenv('GST_DEBUG_DUMP_DIR_DIR', '/tmp')
try:
    import gi
except:
    pass
try:
    from gi.repository import Gtk
except:
    sys.exit(1)
try:
    import pygst
    pygst.require("0.10")
    import gst
except:
    pass

# creating a basic transcoder class
class Transcoder:
    def __init__(self):
        self.pipeline = gst.Pipeline("TranscodingPipeline") # creating overall pipeline object

        # Creating GStreamer filesrc element and sets it to read a specific mp3 file
        self.filesrc = gst.element_factory_make("filesrc", "filesrc")
        self.filesrc.set_property("location", """/home/cschalle/Music/tok.mp3""")
        self.pipeline.add(self.filesrc) # add this first plugin to the pipeline object

        # Use highlevel decodebin2 element to choose which GStreamer elments to use
        # for decoding automatically
        self.decoder = gst.element_factory_make("decodebin2", "decoder")

        # Connect to signal that will let us know that decodebin2 got a pad we can connect
        # to which has the decoded media file on it
        self.decoder.connect("new-decoded-pad", self.OnDynamicPad)
        self.pipeline.add(self.decoder)

        # create an audioconvert element to convert bitrate if needed
        self.audioconverter = gst.element_factory_make("audioconvert", "audioconverter")
        self.pipeline.add(self.audioconverter)

        # create audioencoder, in this case the Vorbis encoder
        self.audioencoder = gst.element_factory_make("vorbisenc", "audioencoder")
        self.pipeline.add(self.audioencoder)

        # create ogg muxer to hold vorbis audio
        self.oggmuxer = gst.element_factory_make("oggmux", "oggmuxer")
        self.pipeline.add(self.oggmuxer)

        # create file output element to write new file to disk
        self.filesink = gst.element_factory_make("filesink", "filesink")
        self.filesink.set_property("location", """/home/cschalle/Music/tok.ogg""")
        self.pipeline.add(self.filesink)

        # Now that all elements for the pipeline are create we link them together
        self.filesrc.link(self.decoder)
        self.audioconverter.link(self.audioencoder)
        self.audioencoder.link(self.oggmuxer)
        self.oggmuxer.link(self.filesink)

        # set pipeline to playing which means all the connected elements in the pipeline
        # starts pushing data to each other
        self.pipeline.set_state(gst.STATE_PLAYING)

    # create a simple function that is run when decodebin gives us the signal to let us 
    # know it got audio data for us. Use the get_pad call on the previously 
    #created audioconverter element asking to a "sink" pad.
    def OnDynamicPad(self, dbin, pad, islast):
        pad.link(self.audioconverter.get_pad("sink"))

# extremely simple UI using a GtkBuilder UI generated with Glade, just two buttons. 
# One to start transcode and one to run pipeline debug
class SuperSimpleUI:
    def __init__(self):
       self.builder = Gtk.Builder()
       self.uifile = "supersimple-gtk3.ui"
       self.builder.add_from_file(self.uifile)
        self.window = self.builder.get_object ("MainWindow")
       self.window.connect ("destroy", self.dialog_destroyed) # this allows the application
                                                              # to be cleanly killed
       # Call the two buttons in the UI
       self.transcodebutton = self.builder.get_object("transcodebutton")
       self.debugbutton = self.builder.get_object("debugbutton")

       # Connect to the clicked signal on both buttons
       self.transcodebutton.connect ("clicked", self.on_TranscodeButton_clicked)
       self.debugbutton.connect ("clicked", self.on_debug_activate)

      # set window size to avoid it being so small it gets lost on the desktop
      self.window.set_default_size (580, 435)
      self.window.show ()

    def on_TranscodeButton_clicked(self, widget):
        self._transcoder = Transcoder()
        print "transcoding"

    def dialog_destroyed (self, dialog):
        Gtk.main_quit ()

    # this function generates the dot file, checks that graphviz in installed and
    # then finally generates a png file, which it then displays
    def on_debug_activate(self, widget):
        dotfile = "/tmp/supersimple-debug-graph.dot"
        pngfile = "/tmp/supersimple-pipeline.png"
        if os.access(dotfile, os.F_OK):
            os.remove(dotfile)
        if os.access(pngfile, os.F_OK):
            os.remove(pngfile)
        gst.DEBUG_BIN_TO_DOT_FILE (self._transcoder.pipeline, \
        gst.DEBUG_GRAPH_SHOW_ALL, 'supersimple-debug-graph')
        # check if graphviz is installed with a simple test
        try:
            dot = which.which("dot")
            os.system(dot + " -Tpng -o " + pngfile + " " + dotfile)
            Gtk.show_uri(None, "file://"+pngfile, 0)
        except which.WhichError:
            print "The debug feature requires graphviz (dot) to be installed."
            print "Transmageddon can not find the (dot) binary."

if __name__ == "__main__":
hwg = SuperSimpleUI()
Gtk.main()

The first thing happening in the file after importing the basis system classes and the which.py tool, is that we set the ‘GST_DEBUG_DUMP_DOT_DIR’ environment variable. When you set this value, GStreamer will be able to at any time dump the pipeline and elements to a ‘dot’ file, which can be turned into a nice looking png by the graphviz command line tool (should be available in most distributions).

Next I import GTK and GStreamer, as you see I don’t yet use the gobject introspection version of GStreamer as that is not fully working yet, but I plan to try to port this simple application to GStreamer 1.0, in which gobject introspection will be the supported way of using Python.

Next is setting up the GStreamer pipeline. You always start by creating a pipeline object, consider this the canvas onto which you will paint the GStreamer streaming pipelines. The next step is to assemble all the GStreamer plugins we want to use in the application. First I create a filesrc object pointing to the file I want to transcode, be sure to point that to a file of your own if trying this application. Next is creating the decodebin2 element. Decodebin2 is one of a set of high level elements in GStreamer, called bins, which contains a wide range of plugins inside. These high level elements are there to make things a lot simpler, and in the case of decodebin2 it will automatically put together the plugins needed to convert your incoming file to raw audio and video (or just demux the file). This means your input doesn’t need to be a mp3 file, like I used, as decodebin2 will reconfigure itself to handle any file you throw at it. After this I create a series of elements to enable me to encode the data into a Ogg Vorbis file. I am doing that to help explain how elements are stringed together, but there is another high level element, encodebin, which I could have used instead. Transmageddon uses encodebin in its git version.

Once all the elements are created you can think of them as boxes spread around on your pipeline canvas, but in order for GStreamer to know how you want to connect them together you need to link them together, as you can see I do with statements like ‘self.filesrc.link(self.decoder)’, which connects the filesrc element I created with the decoder element.

The one special element here is decodebin, which being a dynamic element I need to link it once the pad found signal is fired. Also to link I need to request a compatible pad from the element I am linking with, in this case the audioconverter element.

The last part of the GStreamer setup is setting the pipeline to playing state, which is the state where the pipeline is running. While not a big concern in this very simple application, dealing with state changes in GStreamer is going to be one of the major items you look out for. The GStreamer plugin writers guide contains a chapter discussing the basics of the four states "NULL", "READY", "PAUSED" and "PLAYING". Your pipeline (and all elements) always start at Null state and will go through each of the other stanges to reach Playing. So while we only set state to PLAYING in this simple application, GStreamer will in the background go through READY and PAUSED. The reason the intermediary states matter is because certain things happen at each, so for instance if you want to do some analysis of a file before starting to run your pipeline fully you want to be in PAUSED state as GStreamer will then start pulling the initial data through the pipeline and thus allow you to get information from your elements about the stream or file. One important thing to keep in mind as you develop more advanced applications is that the individual elements can have a different state than the pipeline, but when the state of the pipeline changes it will change the state of the plugins along with it, so you never want your pipeline to be more than one level lower than any of your elements, as that will cause the element to jump down to that state and thus lose the negotiation and information it had assembled.

I am not going to go into a lot of detail about the GUI, it is a very simple GTK user interface built using Glade, and hooked up using the GTK3 gobject introspection bindings. If you got any questions about it post a comment and I be happy to talk about it. What I want to talk about instead is the on_debug_activate function. I wrote this for Transmageddon, but my hope is that it will be useful for anyone writing a Python application with GStreamer (and I guess it shouldn’t be to hard to port to another language). It will allow you to add a menu entry or button in your application that outputs a png file, like the one you see below, which gives you a nice full view of the pipeline used by GStreamer. Especially if you use things like decodebin2 and encodebin, or have a lot of code dynamically adding/removing elements, it can be really useful to see what pipeline ended up being used. And if you have elements that you created, but forgot to link inn, they will appear as orphaned boxes in the file, allowing you to detect such issues. The important thing to remember is that it needs the graphwiz application to be installed on your system and available in the executable path.

Image of GStreamer pipeline
Generated GStreamer pipeline png

Anyway, I hope this has been useful and I plan to post and updated version of this simple application, ported to use encodebin and GStreamer 1.0.

Google and Patents

As seen on Slashdot and other places there has been quite a lot of public discourse recently in wake of consortiums consisting of Microsoft and Apple among others buying the Novell and Nortel patents. This caused Google through its Senior Vice president, David Drummond, to call the use of these consortiums to buy patents anti-competitive. And he is absolutely correct. In the Zdnet article they point to a tweet by Microsofts Brad Smith who counters that they offered Google to bid with them. Well as I pointed out when blogging about the Novell patent sale, the problem here is that these patents are needed to deter lawsuits from Microsoft, and co-owning some patents from Microsoft doesn’t really achieve that.

Another horrid attempt at a rebuttal to Google is from John Gruber who starts by trying to equate the patent problem with Google undercutting Microsoft on price. Why even though he partially admits the US patent system is broken later on, he tries to say that Microsoft attacking Google with bogus patents is the moral equal to Google competing with Microsoft through undercutting them on price…/facepalm.

And he then tries to be clever and rhetorically ask “But what exactly does Google need to defend against, if not actual patents Android actually violates?”. Well the answer to that is that Google needs its own pool of bogus patents as it is the only way to protect yourself from other peoples bogus patents. But what John seems to forget is that two wrongs doesn’t make a right, and unless the US congress at some point decides to actually do something that helps the US economy and drop the stupidity that is software patents, then innovating companies will continue to need to waste money and time on software patents, so protect themselves form the attempts of the market incumbents to abuse the patent system to shut down their competitors.

For those wanting to read up on how damaging the current software patent regime is I recommend this article from the Economist called Patents against Prosperity and this blog by Craig Hockenberry called the Rise and Fall of the Independent Developer.

The sad part though is that by the time Google manages to build up their patent arsenal to protect themselves, I am sure they will have managers who decide that in order to protect the interest of Google shareholders, Google should start to favour software patents, just like Bill Gates turned coat on the issue once they realized that while patents lawsuits would cost them a fortune, they could use it to kill of a lot of potential competitors.

First GStreamer 0.11 out, GStreamer 1.0 looming on the horizon

GStreamer maintainer Wim Taymans just released the first 0.11 development release of GStreamer. The 0.11 development series will lead up to the long awaited GStreamer 1.0 release!

The changes from 0.10.x to 0.11 might seem quite technical and obscure to most, with items such as reworked buffer memory management, arbitrary buffer metadata and integrated bufferpool management being among the advertised features, but all these changes are made to help GStreamer make significant leaps forward in terms of integration with hardware codecs like VAAPI and VDPAU and of course hardware codecs on embedded platforms like ARM for instance the OpenMax IL API. There are also a lot of important performance improvements, which will make applications like Totem and Banshee more snappy to use, but you probably will see the biggest improvements in applications like PiTiVi who relies on more complex pipelines and thus more complex pipeline negotiations. For devices which got more constrained CPU resources, like various embedded systems, these performance gains should also be very noticeable.

At Collabora we are putting a lot of effort towards GStreamer 1.0, most visible through letting Wim put most of his work hours into it, in collaboration with our partners at Texas Instruments. GStreamer is not just another open source project for us at Collabora, it is something we are truly passionate about. The open source software ecosystem can not compete with proprietary systems unless we have a top notch media framework and with GStreamer we are providing exactly that. Ever since the release of GStreamer 0.10 the project has gone from strength to strength, and when 1.0 gets released later this year it will be another major milestone towards world domination :)

For those interested to learn more about GStreamer 1.0 you have two good opportunities coming up, Collabora’s own Wim Taymans and Edward Hervey will be doing a talk about GStreamer 1.0 at the Desktop Summit on 8th of August. And Wim will also be doing a keynote speech about GStreamer 1.0 at the GStreamer Conference 2011 in October. So I hope to see you there.

I am also working on an interview with Wim Taymans about GStreamer 1.0 so if you have any questions you would like me to include, feel free to add them to the comments section of this blog post.

We will also be organizing some 0.11 hackfests online where people like Wim, Tim and Edward will be online to answer porting questions and the community can work together to port all important plugins to 0.11. There is some early stage porting documentation to be found here.

For now, go to the GStreamer website and grab the 0.11 tarballs and give them a spin, and if you have any questions, remember you are always welcome in #gstreamer on irc.freenode.net.

Weekend hacking

Spent some time this weekend hacking on Transmageddon. Fixed various small bugs and UI issues that I had punted up until now for the UI. For instance with latest git when you create a pure audio file it doesn’t automatically get the suffix .mp3, which is nice in the cases when you are not creating a mp3 file :) And if you put aac into a quicktime container the file gets named .m4a instead of .mov.

Also started looking into the issue of how to handle multiple audio streams in the file being transcoded. Currently all streams gets transcoded to the same chosen format if the container format support its, if the container only supports 1 audio stream you get one by random. This is not ideal :)

Ended up filing this bug with a request for how we can improve the GStreamer API to make handling such things easier for application developers. Discoverer, uridecodebin and encodebin makes a lot of things a lot easier, but for handling files with multiple streams of the same type I think we still need some improvements.

On the path to GStreamer 1.0

GStreamer maintainer and code god Wim Taymans just posted an update on the the progress of GStreamer 0.11 to the GStreamer development mailing list. For those interested in learning about the new features coming in GStreamer 1.0 this email (along with the previous update) is must read material.

In addition to the updates on the core coding work and important notice from Wim in his email is that the very first 0.11.x release will happen this week, so that you have a snapshot release to start playing with. GStreamer 1.0 is moving forward at a fast pace, so be prepared :)

Hopefully we can do a GStreamer 1.0 release part at the GStreamer Conference 2011!

GStreamer Conference 2011 update

Just sent out this little email with some updates on the GStreamer 2011 Conference. Planning is progressing and a sponsorship leaflet is now available for those interested in sponsoring the conference. The call for paper deadline is also slowly but surely approaching, so anyone who wants to do a talk please send in an abstract before the 1st of July this year.

For everyone else, just register for the conference and set aside October for GStreamer and Prague :) As always details can be found on the GStreamer Conference 2011 website.

Edward Hervey talks about GStreamer Editing Services

A second Collabora talk from the Meego conf is now online. Edward Hervey did a talk about GES, the GStreamer Editing Services, which is a library that is part of GStreamer which makes video editing and encoding a lot easier. It it part of GES that I use in the git version of Transmageddon and there is also a plan to port PiTiVi to use GES.

Edwards talk focuses on using GES on embedded systems, as it was done at the MeeGo conference, so if you are interested in video editing be it on the desktop or on an embedded system be sure to check out the talk. You can find Edwards talk here on the Meego website.

OpenOffice vs LibreOffice – the next chapter

Been seeing with interest the latest moves around Open Office. While a lot of people see it as almost a direct attack on Libre Office, to me personally it seems like a clumsy result of Oracle trying to ditch OpenOffice without frustrating their main OpenOffice business partner, IBM. Due to having the Lotus Symphony suite based on OpenOffice under a special license from Sun/Oracle, I wouldn’t be surprised if switching to the pure LGPL Libre Office seemed painful to them. And thus the idea of an Apache licensed OpenOffice must have seemed endearing.

Personally I hope people stick with LibreOffice and build upon their existing success. Chasing a big company like IBM might seem tempting, but big companies change their mind and change priorities all the time, just look at Nokia, so if you have something viable without a big company involved, stick with it, and let the big company contribute on your terms if they want, as it will then have the ability to stay around even when the big company goes elsewhere.