Just another GNOME Blogs weblog

Warning, long post
With a blogging rate < 1 post/year you will forgive me if the single one I write is a bit long. If you're impatient jump directly to the !#!#PROTECT0#!#! ;-).

What happened to gleffects?
You may have noticed the astonishingly cool super awesome effects I wrote for Cheese during my Summer of Code never made their way into Cheese itself.

Writing the effects with GLSL was and still is super fun.
It’s amazing what you can do even with a cheap gpu.
A couple of months ago, thanks to my lovely nvidia gpu heating up to the point of desoldering itself from the mainboard and making my laptop unusable ((Maybe some day I’ll tell you how I did solder reflowing and temporarily fixed the laptop with a heatgun)), I ported most of the effects to work on my little i915 based netbook. It’s quite satisfying to see convolution filters (gaussian blur, sobel edge detector, glow) process 640×480 textures at a nice framerate on such a small and underpowered device.

I believe we definitely need to take advantage of this big power in GStreamer and we’re slowly going towards the right direction.

But…
As cool as it can be, at the moment, gst-plugins-gl is still little more than a hackish proof of concept. It’s difficult to use in an application especially if you want to have mixed gl and normal buffers. Each frame needs to be uploaded to the video memory, processed with GL and downloaded back into ram for further gstreamer processing (e.g. video recording). So you need dedicated code paths that wouldn’t be needed if GL video was a first class citizen in a gstreamer pipeline.

The thing is even worse if you want to use GL in the UI, like we’re finally doing in Cheese. Gst-plugins-gl and Clutter run in separated threads, they use different GLX contexts and although there are ways to share textures (either with glxcontext sharelists or with texture from pixmap) between the two, they are tricky and fragile.

That and, most of all, the little time my studies leave me, kept the effects far away from Cheese all this time.

What changed?
Nothing.

So what?
Some days ago, during one of those moments after an exam when you feel so bored to suddenly have nothing to do but don’t want to start studying for the next one and desperately look for something to hack on, I noticed warptv.
It’s an effect we had in Cheese since the beginning. It applies a time dependent distortion to the image. A distortion, nothing that special, just like some of the funniest filters from gleffects, without gl.

So I went to look at the code and found out it was pretty simple to reimplement all distortion filters in gleffects using GstVideoFilter class.

No gpu, won’t it be slow?
While the shaders in gst-plugins-gl are able to calculate new coordinates for each pixel on the fly, to obtain acceptable speed on the cpu you have to precalculate a distortion map in an outer loop.
That’s all, the end result is the same and probably the cpu one is even faster because you don’t have the texture upload overhead.

There is a minor drawback ((Two actually, the gpu also does out of bound pixel clamping for free)) though: with the gpu you work on floating point coordinates and if your transformed pixel falls at fractional coordinates the hardware will calculate its color interpolating from the four nearest pixels.
With the cpu no one helps you and you have to either map the transformed fractional pixel to the color of its nearest neighbour (fast but jagged and ugly results) or implement bilinear interpolation yourself (nice but slow).

Cool, where I can see the results?
At first I ported all distortion filters from gleffects to a new plugin called cheeseeffects but soon discovered that Thiago already did something similar with geometrictransform plugin. So ported all the filters ((Minus the ones already there: squeeze, there called pinch, and twirl)) to his base class, fixed some of my old math and made effect parameters customizable. Unfortunately geometrictransform only does nearest neighbor interpolation, but I plan to port the bilinear interpolation routine I wrote in the first iteration sooner or later.

Done with distortion ones I also ported the color lookup table effects (heat, sepia, xpro and xray) into a new plugin, coloreffects.

I said see, where’s the screencast?
Okay okay, here’s your obligatory screencast!

Effect live previews from new Cheese effect selector

As you can see the effects look almost like the gl ones, but there is no gl involved. No need for a powerful gpu, no need for good video drivers, no need for GLSL! The screencast is taken on my netbook directly from the new effect selector Yuvi wrote for his Summer of Code and the low framerate is caused only by the recording app. The effect previews run pretty smoothly here.

Where can I get it?
Everything, thanks to Sebastian Dröge for the super quick review, is already in GStreamer Bad Plugins, you can test it if you want from today’s prerelease tarballs and will be for sure in the new Cheese.

PS
As you may have noticed there is a new Donate button in the sidebar.
Quite often someone suggested me to put one, for my work in Cheese, but never liked the idea that much…
Nonetheless, I’m an unemployed student and all the time I spend hacking in free software, albeit fun and exciting, is time subtracted from my studies.

Also as I said earlier, since some month I’m forced to work on a netbook as my laptop is dead and cannot afford a new one at the moment. I definitely wouldn’t mind some help there 🙂

So, if you appreciate my work, feel free to donate 😉
Thank you!

Well, not so lazy… I’ve spent a couple of hours looking into this but didn’t find a solution.

Is there a way to set default size of a widget (or better just of a GtkDrawingArea) without limiting its minimum size?

It seems that the only way to have a drawing area of the size I’d want is to set a size request.

Thanks!

Hey I’m back on planet gnome isn’t it cool? Many thanks to Lucas and Jeff!

Some of you may already know that I’ve been laptop less in the past three months. Since about mid September display started to behave odd: it randomly turns off the lamps until I close and reopen the lid, then after about half a minute it turns off again, then again on, and so on. Sometimes it even starts blinking like a crazy stroboscope 😀 .
Long story short, here is a little video of the crazy display. Awesome, isn’it?

Unfortunately this little issue is a bit difficult to reproduce, sometimes it does it the whole day, and believe me, it’s quite irritating, sometimes it doesn’t for several days… Sometimes it does it only early in the morning (cold lamps?).

I contacted HP Support and they argued it could be a motherboard issue.
A motherboard issue, at least in Italy is under a 2-year pick-up-and-return warranty.
So they sent it to their repair center (actually an external one: A-NOVO Italia, Saronno) and sent it back after about two of weeks with a nice HP headed sheet stating they didn’t find any issue at all and just restored the operative system (a laptop coming with a motherboard issue and they did just that?). They also installed and ran a couple of benchmarks and other test software that said everything was ok, and I guess nothing more.

Guess what… display started blinking again 2 days after.
I contacted again HP, already a bit upset, I talked with another technician and he said his colleague was surely wrong. I had no motherboard issue but just some lamp or hinge one. It was covered by one year warranty with the producer and one with the seller.

Ok, I then went to the seller and told them about the previous story.
They sent the laptop to the very same repair center with a long description saying that it was a bit an unpredictable issue, that it was just been there some week before and suggesting to test it carefully this time. It was the December 1st.

Three days ago I was called to take back the laptop. Same HP headed sheet, all tests were successful, no issue found. Oh, they did a BIOS update, damn you receive a laptop for repair twice with the very same issue and you just run some test and do a BIOS update? Seriously, a BIOS update?!

Needless to say screen started blinking again soon. You can see it in that video taken a couple of hours ago and I can see it in front of me right now.

I don’t know if I feel more upset or frustrated… still I cannot believe how a repair center can suck this way… they didn’t even try to replace the lamps! they didn’t even unmount the display to check the cables!

Hey! Summer of Code just ended! since Monday we’re in Final Evaluation phase.
This means I can finally take some vacation and leave coding for a while 🙂 !
But, most of all, it means that it’s time again for not one, not two, but four screencasts!!

I’m sorry for not posting anything for a while but, needless to say I don’t like blogging :P, I was really busy with the merge. Julien Isorce, another gstreamer developer, was working on his own branch of gstreamer-gl since some time and recently David Schleef decided to merge it with cvs trunk.
He added a lot of great things like glsl colorspace conversion, win32 support, a working glimagesink and lot more. Unfortunately my effects didn’t work with the new code so I had to work hard to make them work again.

But let’s stop talking about this boring stuff, now almost everything is fine so… get ready for the show 😛 !

New Effects

New Effects

Not too much on this side..

• a Glow Lighting effect that some of you requested. I tried to resemble gimp SoftGlow filter, do you like it?
• a SinCity inspired effect that desaturates everything but red shades… don’t know if you agree but I love it!
• better curves for Sepia and Cross Processing effect, maybe the latter still needs some tuning (too light blue on dark tones).
• a Square distortion effect… ehm, no.. it was already there 🙂 but that’s definitely my favourite one!

New Filters

These are still young and don’t work quite well yet, but I’m almost there!

GstGLPixbufOverlay

Takes an image filename through a property and alpha blends it with the video stream.
It could be nice with Cheese to add funny frames, or just to add a red circle to displayed video while recording.

GstGLDifferenceMatte

This one saves a video frame of the background and replaces it with an image.
It’s really hard to make it work properly, it tries to guess the background thresholding the difference between current frame and saved one.
To get a good result it needs a uniform background, better with a uncommon color, a low noise webcam (i tried to remove noise dependency with some gaussian blur), etc.
Any idea to improve it? Probably RGB is not the right colorspace to do this thing, maybe using HSL or HSV and thresholding just hue could help to get rid of luminance dependency. I’ll give it a try.

Clutter Experiments

Clutter live preview

This is my last experiment with rendering from gstreamer gl into a ClutterActor.
It uses XComposite to redirect glimagesink stream offscreen and Clutter Texture from Pixmap support (thanks Andy Wingo for the idea) to turn it into an Actor.
It’s still young and experimental but it could be a good starting point for a clutter based effect selector for Cheese.

Great, but where is the code?
There is no release yet but you can test it if you want from the git repository git://diracvideo.org/git/gst-plugins-gl.git
Take a look at tests directory.
It is known to work for sure on linux, windows and probably osx (through X).
It needs a recent gpu that supports FBOs (mandatory) and GL Shading Language (optional, needed for the effects and accelerated colorspace conversion).

Thanks
That’s all for now.
A big thanks to everyone who helped, supported, encouraged me during this Summer of Code, Daniel Siegel, David Schleef, Julien Isorce, the guys from #gstreamer, #clutter, #gnome-it, ##OpenGL, the GNOME Community, everyone!
Thank you, it’s been a great time!

Hey, it’s already July 7th (well, almost 8th now), time for midterm evaluations for SoC students!
So how could I leave you without the mandatory screencast? It’s been a bit painful to overcome my shyness and I felt a bit stupid recording myself.. but that’s it, hope you’ll enjoy!

Yay! that’s me!

It’s pretty much self-explanatory, just look at it and have fun!

I’m quite satisfied with the face distortion effects, aren’t they awesome (as Daniel would say 😉 )?
I still have some doubt, though, about the sepia effect. It can be obtained just with the gimp curve tool, so if you have a better one feel free to send me the curve file 😉 (or any other nice curve file that you have)!

I’m really looking forward to hear your comments and suggestions!

GUADEC Rants..
It seems that I won’t be able to be at GUADEC with all of us, it’s such a pain to read your posts and cannot be there..
I obtained the travel sponsorship but I really needed an answer for a couple of questions before being able to buy tickets. I asked Baris but he someway disappeared since last wednesday, I never seen him again on irc and he stopped replying my mails (and someone else’s too as far as I know).
So time passed, and I’m still here waiting for his reply… I can understand he should surely have been over busy within past week, but it was too bad that he was the only one to contact for sponsorship questions.
There is still a (very) little chance to take a last minute flight so Baris, if you’re reading this, just mail me, comment, ping me on irc!

Hey all! As you might have noticed I’m not too much into blogging, my last post is more than 2 weeks old..
Anyway, I’m supposed to post a weekly status report to the gnome-soc list, so why not to take it as a chance to keep you all updated about my work? So, expect at least a post per week from now on, around each monday 😉

Brief summary of first week

I didn’t code too much the first week since I’ve been very busy with last lessons at the university.
Besides it’s not that I didn’t do anything, I managed to start some basic work with gstreamer gl plugin. I began to extend the gstreamer gl library with some GLSL shader support, basically I’ve written a couple of GObject classes to abstract from gl primitives and use them easily.
I’ve also written a little gtk test app that shows video images with xvimagesink and xoverlay and select effects with a combo button. Nothing too complicated but more than enough for what I need at the moment.

2nd Week Report

Within last week I kept on working on gstreamer gl library, I’ve finished and committed the first draft of a shader abstraction library, it’s capable of creating program and shader objects with usual gobject methods, compile shaders, attach them to a program, link and attach them to GL pipeline.
At the moment, so, it simply wraps the correspondent gl methods, take care of error handling, and clean up everything when program or shader ref count reaches 0.
The gobject abstraction, though, will simplify, for example adding support for ARB programs (asm shaders) and use them with the same api of GLSL ones.
It would also allow to have convenience methods like gst_gl_shader_add (not yet implemented) to have a shader compiled, attached and linked with just one call, or to keep track of multiple shaders attached to the same program object so that it would be easy to release them all at once. Furthermore the use of gobject makes code clean, and easier to read and debug.
There are still a couple of things I’m not quite sure about:

• How to handle errors?

GLSL shaders are designed to resemble C programs, hence they have to be compiled and linked before attaching them to the GL state. This process could rise errors, warnings and info messages that can be queried and are extremely useful for debugging. How should my library handle them? At the moment I’m setting a GError with compilation and linking messages and leaving to the user (the GstGLFilter element) the task to handle them. Should I use g_error, g_warning and g_message instead? Or should I use GstInfo debug macros directly from the shader objects?

• Ehm.. I forgot the others while writing about the previous 🙂

I’ve also started to write the real GstElement that all my summer project is about, GstGLEffects (the name is not definitive).
It’s a subclass of GstGLFilter written taking David’s GstGLFilterExample as a model. I didn’t commit this yet, but it’s already capable of using the new shader abstraction library and can be used as a starting point to develop the effects.
It maps video texture to a GL quad (at the moment switched right to left to resemble looking at a mirror when used with webcam stream, but I plan to set this with a property sooner or later).
Effects are being selected with a enum property that sets an effect_changed attribute and binds an effect function pointer to the right effect. This is needed because gl functions have to be called within the basetransform filter methods where we can be sure there is a valid gl context and display is locked. This has a positive side effect to allow to set effects even when the pipeline is paused (and of course when is playing). There is also no need to rebuild the pipeline every time like Cheese does at the moment since we just have one element that does all the work.
To concentrate on effects developing at the moment only one effect at the time can be used (since it was simpler to implement) but I plan to add support for multiple effects in the future.
I also began to play a bit with some effect, mainly distortion ones, but they are still at an early stage so stay tuned during next weeks to see more!

I think that’s all for this week, I surely have forgotten something and as always I didn’t manage to be concise 🙁 I promise to post more often so each post will be shorter next times 😉

Hey all, I’ve been added to planet gnome! (thanks lucasr 😉 )
Since most of you shouldn’t know anything about me, here is a brief introduction.

My name is Filippo Argiolas and I’m a third year Physics student at University of Cagliari (Italy).
I’ve been a gnome user for quite some time and recently I’ve started to actively contribute, well… when my studies left me some spare time …
I’ve written some minor patch here and there and maybe some of you already know the little game I’ve written some time ago.

It’s been a while since I’ve been accepted to Google Summer of Code 2008 😀 , I wanted to blog about it before but I’ve always been so busy at university and never had the time.
By the way, it’s never too late, so let’s talk a bit about my summer project.

I’ll work on a set of opengl based gstreamer elements to improve Cheese with some new hardware accelerated effects.
Most of my work will involve adding GLSL Shaders support to gstreamer gl, and creating some fragment shader to take advantage of modern graphic adapters’ programmable pipeline. This will hopefully end up with a bunch of effects, from distortion to color transforming and more artistic ones, with almost no cost for the cpu.
Being based on gstreamer, most of the stuff won’t be restricted to Cheese and will be reusable wherever gstreamer is used to manage video data.

You can check a more detailed description in the abstract

As I said in the previous post (not yet aggregated in the planet), I’ve opened this blog to keep you updated of my progresses with my summer project, and, most of all, to get feedback, suggestions and, maybe, help. So feel free to leave your comments here!

Please forgive for being so long, I’m still new to bloggin and I’m practicing to get some more synthesis skill.

Before to leave I’d like to thank everyone who gave me this great chance to pass the summer working on something such fun, most of all Daniel Siegel, my mentor, and David Schleef, for letting me work on gstreamergl! Thank you guys!

Hey all, finally I’ve opened a weblog.. that’s my first blog post ever!

I never really liked the whole idea behind blogs, and I always said I won’t ever have one.. but time’s up to put aside all my bias about it and consider the positive things related 🙂

I’ve been accepted to Google Summer of Code 2008 😀 and this blog will be a quick and easy way to interact with freesoftware community, get feedback, ask for suggestions, beg for help, and so on..

A bit about myself:
My name is Filippo Argiolas, I’m 24 and I study Physics at University of Cagliari.
My summer job for google will consist of writing a gstreamer plugin to extend Cheese with some funny opengl effect.
More detailed infos about it will follow soon!