Broadway adventures in Gtk4

One of my long running side projects is a Gtk backend called “Broadway”. Instead of rendering to the screen this backend creates a HTTP server that you can connect to, and then exposes the UI remotely in the browser.

The original version of broadway was essentially streaming image frames, although there were various ways to optimize what got sent. This matches pretty well with how Gtk 3 rendering works, particularly on Wayland. Every frame it calls out to all widgets, letting them draw on top of a buffer and then sends the final frame to the compositor. Broadway just inserts some image delta computation and JavaScript magic in the middle of this.

Enter Gtk 4, breaking everything!

However, time moves on, and the current development branch of Gtk (which will be Gtk 4) has completely changed how rendering works, with the goal of doing efficient rendering on modern GPUs.

In the new model widgets don’t directly render to a buffer. Instead they build up a model of how the final result should look in terms of something called render nodes. These describe rendering as a tree of highlevel operations. The backend (we have software, OpenGL and Vulkan backends) then knows how to take this description and submit it to the GPU in an efficient way. This is somewhat similar to the firefox WebRender project.

Its would be possible to implement the broadway backend by hooking up the software renderer, letting it generate a buffer and then send that to the browser.  However, that is pretty lame!

CSS comes to the rescue!

Instead I’ve been looking at making the browser actually draw the render nodes. Gtk defines a lot of its UI in terms of CSS these days, and that means that the render nodes actually are very close to the CSS rendering model. For example, the basic drawing operation are things like rounded boxes with borders, shadows, etc.

So, I was thinking, could we not take these render node and turn them into actual DOM nodes with CSS styles and send them to the browser. Then every frame we can just diff the DOM trees, sending the minimal changes necessary.

Sounds crazy right? But, it turns out to work pretty well.

Check out this example page which I created with the magic of “save as”. In particular, try zooming into that page in the browser, and play with the developer tools inspector to see the nodes. Here is a part of it zoomed in:

The icons and the text are not CSS, so they don’t scale, but look at those gorgeous borders, shadows and gradients!

Entering the 3rd dimension!

Particularly interesting is the support in Gtk for general 3D transforms. This maps well to the CSS transform on the browser style.

Check out this example of a spinning-cube transition. If you open up the browser inspector you can see that each individual element in the cube is still a regular CSS box.

Some technical notes

If you look at the examples above they all use data: uris for images. This is a custom mode that lets “screenshots” like the above work. Normally broadway uses blobs for the images.

Also, looking at the examples they seem very heavy in terms of images, as all the text are images. However, in a typical frame most of the render tree is identical to the previous frame, meaning any label that was used in the last frame need not be sent again. In fact, even if it changes position in the tree due to a parent node changing (scrolling, cube-switching, etc) it can still be reused as-is.

However, text is clearly the weak point in here. Unfortunately HTML/CSS has no low-level text rendering APIs we could use. I’m considering generating a texture atlas with pre-rendered glyphs that can be reused (like CSS sprites) when rendering text, that would mean we will have to download less data at least. If anyone has other ideas I would love to hear about it.

Introducing Gthree

I’ve recently been working on OpenGL support in Gtk+, and last week it landed in master. However, the demos we have are pretty lame and are not very good to show off or even test the OpenGL support. I’ve looked around for some open source demos that used modern GL that we could use, but I didn’t find anything that we could easily use.

What I did find though, was a lot of WebGL demos that used three.js. This looked like a very nice open source library for highlevel 3d rendering. At first I had some plans to bind OpenGL to gjs so that we could run three.js, but this turned out to be a hard.

Instead I started converting three.js into C + GObject, using the Gtk+ OpenGL support and the vector/matrix library graphene that Emmanuele has been working on recently.

After about a week of frantic hacking it is now at a stage where it may be interesting for others. So, without further ado I introduce:

https://github.com/alexlarsson/gthree

It does not yet support everything that three.js can do, but it does support a meshes with most mesh matrial types and lighting, including a loader for the json model format of thee.js, which means that it is minimally useful.

Here are some screenshots of the examples that ships with the code:

Screenshot from 2014-10-24 15:04:47
Various types of materials
Screenshot from 2014-10-24 15:10:00
Some sample models from three.js examples
Screenshot from 2014-10-24 15:31:40
Some random cubes

This has been a lot of fun to work on as I’ve seen a lot of progress very fast. Mad props to mrdoob and the other three.js developers for creating three.js and making it free software. Gthree is a huge rip-off of their work and would never be possible without it. Thanks also to Emmanuele for his graphene library.

What are you sitting here for, go ahead and play with it! Make some demos, port some more three.js features, marvel at the fancy graphics!

Broadway on openshift

I spent some time this week making a clean build of Gtk 3 with the broadway backend for deploying on openshift. It works suprisingly well, given that I’m in sweden and the openshift servers are somewhere in the US:

[youtube]https://www.youtube.com/watch?v=cyHTAGoEuL8[/youtube]

I made the video instead of linking to the app itself because only one person at a time can use the actual app. However, you can easily create your own openshift module based on this as I made the openshift code availible at gitthub. Just clone that and push to your diy based openshift app. Play with it! Build your favourite gtk3 app! Share it!

It lacks authentication/encryption and there are some minor issues, but it proves that broadway works reasonably well even over the internet.

Broadway Multi-process support

I just landed support for multiple processes in broadway. The way it works is that we run a process (broadwayd) similar to the X server, which all broadway apps talk to. Additionally it has a webserver built in that the browser connects to.

This allows (for instance) broadway processes to start other broadway processes. Basically the webpage turns into something like an X session.

This is best shown with a demo:

[youtube]http://www.youtube.com/watch?v=hhMFD3ZCrIc[/youtube]

 

Moar windows themes!

Last time I blogged about the Gtk 3 windows theme we had just landed the initial support. It kinda worked but didn’t look quite right. Since then the css machinery has gotten a bit more capable, and I recently found some time to work on this again.

Testing had revealed that the theme was totally broken on Windows XP. This had two main causes; first of all there was some bugs in the Win32 theme APIs on XP when rendering to surfaces with alpha, and secondly the css file used some windows theme parts that only existed in Vista and later. I added workarounds for the alpha bug and introduced a new css file that is used on XP (although most of the css is shared). So, now XP support is working.

I also went through all the widgets, fixing a lot of details in how they render and adding theming for some less common widgets.

Here is how gtk3-widget-factory looks now:

Windows 7
Windows XP

Not bad for 998 lines of css.

This is all in the new Gtk+ 3.4.0 release. We hope to have window binaries out for it soon.

Broadway update 3

This week saw some new updates of the broadway backend. We now have a in-browser window manager for the non-toplevel window mode, and the backend now support a bunch more features.

I don’t want to bore you with technical mumbo jumbo though. Lets see some video instead! (Original source availible here)

[youtube width=”512″ height=”384″]http://www.youtube.com/watch?v=AO-qca9ddqg[/youtube]
This will be the last update in a while, as I need to spend time on other things. The code is in a pretty good shape though. There are still things to do, but most things work.

Update: Tested this with safari on OSX, and it worked too. Also we now have nicer browser-side window decorations.

Gtk+ HTML backend update

The last few days I spent fixing up some more details in the new HTML5 gdk backend. Not everything is supported yet (keyboard input in particular is very weak), but much more things work now. Even thought the backend is not of production quality it is now good enough that I think its interesting for a larger audience to play around with. So, today I merged the branch into the Gtk+ master branch (i.e. what will be Gtk+ 3.2).

The new multi-backends setup in Gtk+ 3.0 makes this much easier to test. All you have to do is build Gtk+ with –enable-x11-backend –enable-broadway-backend.This will give you a normal X11-based Gtk+ where you can enable the broadway backend at runtime by setting the GDK_BACKEND enviroment variable.

The backend only supports firefox 4 atm, and Mozilla disabled websockets by default, so you have to enable enable websockets for any input to work. However when this is done you can test any app by running:

GDK_BACKEND=broadway your-application&
firefox http://127.0.0.1:8080/

I recorded a small screencast to show this stuff: (webm source file here)

[vimeo width=”512″ height=”384″]http://vimeo.com/21062117[/vimeo]

Note that the recording is using a local connection, it will be slower over the network depending on the network bandwidth.

The return of Client side windows

For a long time now I’ve been working on the client side windows branch of Gtk+. By now it is mostly feature complete when it comes to normal use. However, one of the drivers of client side windows and the initial reason I started working on it is the ability to do offscreen window rendering. The last two weeks I’ve been spending on getting that to work and integrated into the platform.

I think a video says more than a million words here:

[vimeo width=”400″ height=”439″]http://vimeo.com/5126552[/vimeo]
(Original ogg availible here)

This is using the current client-side-windows branch of Gtk+, plus my own gtk-in-clutter code availible in the client-side-window branch of http://gitorious.org/clutter-gtk-copy.

Next up is getting the non-X backends working and getting this merged into master.