Finally, (some of) my effects for everyone

Warning, long post
With a blogging rate < 1 post/year you will forgive me if the single one I write is a bit long. If you're impatient jump directly to the !#!#PROTECT0#!#! ;-).

What happened to gleffects?
You may have noticed the astonishingly cool super awesome effects I wrote for Cheese during my Summer of Code never made their way into Cheese itself.

Writing the effects with GLSL was and still is super fun.
It’s amazing what you can do even with a cheap gpu.
A couple of months ago, thanks to my lovely nvidia gpu heating up to the point of desoldering itself from the mainboard and making my laptop unusable ((Maybe some day I’ll tell you how I did solder reflowing and temporarily fixed the laptop with a heatgun)), I ported most of the effects to work on my little i915 based netbook. It’s quite satisfying to see convolution filters (gaussian blur, sobel edge detector, glow) process 640×480 textures at a nice framerate on such a small and underpowered device.

I believe we definitely need to take advantage of this big power in GStreamer and we’re slowly going towards the right direction.

But…
As cool as it can be, at the moment, gst-plugins-gl is still little more than a hackish proof of concept. It’s difficult to use in an application especially if you want to have mixed gl and normal buffers. Each frame needs to be uploaded to the video memory, processed with GL and downloaded back into ram for further gstreamer processing (e.g. video recording). So you need dedicated code paths that wouldn’t be needed if GL video was a first class citizen in a gstreamer pipeline.

The thing is even worse if you want to use GL in the UI, like we’re finally doing in Cheese. Gst-plugins-gl and Clutter run in separated threads, they use different GLX contexts and although there are ways to share textures (either with glxcontext sharelists or with texture from pixmap) between the two, they are tricky and fragile.

That and, most of all, the little time my studies leave me, kept the effects far away from Cheese all this time.

What changed?
Nothing.

So what?
Some days ago, during one of those moments after an exam when you feel so bored to suddenly have nothing to do but don’t want to start studying for the next one and desperately look for something to hack on, I noticed warptv.
It’s an effect we had in Cheese since the beginning. It applies a time dependent distortion to the image. A distortion, nothing that special, just like some of the funniest filters from gleffects, without gl.

So I went to look at the code and found out it was pretty simple to reimplement all distortion filters in gleffects using GstVideoFilter class.

No gpu, won’t it be slow?
While the shaders in gst-plugins-gl are able to calculate new coordinates for each pixel on the fly, to obtain acceptable speed on the cpu you have to precalculate a distortion map in an outer loop.
That’s all, the end result is the same and probably the cpu one is even faster because you don’t have the texture upload overhead.

There is a minor drawback ((Two actually, the gpu also does out of bound pixel clamping for free)) though: with the gpu you work on floating point coordinates and if your transformed pixel falls at fractional coordinates the hardware will calculate its color interpolating from the four nearest pixels.
With the cpu no one helps you and you have to either map the transformed fractional pixel to the color of its nearest neighbour (fast but jagged and ugly results) or implement bilinear interpolation yourself (nice but slow).

Cool, where I can see the results?
At first I ported all distortion filters from gleffects to a new plugin called cheeseeffects but soon discovered that Thiago already did something similar with geometrictransform plugin. So ported all the filters ((Minus the ones already there: squeeze, there called pinch, and twirl)) to his base class, fixed some of my old math and made effect parameters customizable. Unfortunately geometrictransform only does nearest neighbor interpolation, but I plan to port the bilinear interpolation routine I wrote in the first iteration sooner or later.

Done with distortion ones I also ported the color lookup table effects (heat, sepia, xpro and xray) into a new plugin, coloreffects.

I said see, where’s the screencast?
Okay okay, here’s your obligatory screencast!

Effect live previews from new Cheese effect selector

As you can see the effects look almost like the gl ones, but there is no gl involved. No need for a powerful gpu, no need for good video drivers, no need for GLSL! The screencast is taken on my netbook directly from the new effect selector Yuvi wrote for his Summer of Code and the low framerate is caused only by the recording app. The effect previews run pretty smoothly here.

Where can I get it?
Everything, thanks to Sebastian Dröge for the super quick review, is already in GStreamer Bad Plugins, you can test it if you want from today’s prerelease tarballs and will be for sure in the new Cheese.

PS
As you may have noticed there is a new Donate button in the sidebar.
Quite often someone suggested me to put one, for my work in Cheese, but never liked the idea that much…
Nonetheless, I’m an unemployed student and all the time I spend hacking in free software, albeit fun and exciting, is time subtracted from my studies.

Also as I said earlier, since some month I’m forced to work on a netbook as my laptop is dead and cannot afford a new one at the moment. I definitely wouldn’t mind some help there 🙂

So, if you appreciate my work, feel free to donate 😉
Thank you!