Gtk+ HTML backend update

The last few days I spent fixing up some more details in the new HTML5 gdk backend. Not everything is supported yet (keyboard input in particular is very weak), but much more things work now. Even thought the backend is not of production quality it is now good enough that I think its interesting for a larger audience to play around with. So, today I merged the branch into the Gtk+ master branch (i.e. what will be Gtk+ 3.2).

The new multi-backends setup in Gtk+ 3.0 makes this much easier to test. All you have to do is build Gtk+ with –enable-x11-backend –enable-broadway-backend.This will give you a normal X11-based Gtk+ where you can enable the broadway backend at runtime by setting the GDK_BACKEND enviroment variable.

The backend only supports firefox 4 atm, and Mozilla disabled websockets by default, so you have to enable enable websockets for any input to work. However when this is done you can test any app by running:

GDK_BACKEND=broadway your-application&
firefox http://127.0.0.1:8080/

I recorded a small screencast to show this stuff: (webm source file here)

Note that the recording is using a local connection, it will be slower over the network depending on the network bandwidth.

60 Responses to “Gtk+ HTML backend update”

  1. Alexandre Mazari says:

    Great great stuff !
    Moving applications to the cloud will be a breeze, also application apps could benefits from HTML5 technologies in regard to animations etc…

    I dunno if you read about the SeedKit project but it allow to build UIs in web technologies while accessing from js to native introspected APIs.
    Also the webkit-gtk team is planing on adding the possibility to “inject” a GObject instance into a JS context.

    I can imagine Broadway accessing its backend code by those means for local invocations, instead of going the network way.

    What do you think ?

  2. pankaj says:

    wow awesome stuff, this will probably do away the need to have ssh X forwarding in the future

  3. Have you looked at orbited as a stopgap between now and when websockets are accepted? http://orbited.org/ It uses persistent HTTP connections to emulate a ‘push’ connection (it still polls internally). I believe it also has a websocket interface to make it easy to migrate to websocket once it is stable. I’m not sure it is fast enough but it might be an option.

  4. Tiago says:

    Holy crap… Can you run firefox inside firefox??

  5. Conrad says:

    Seems like this needs a window manager :-)

  6. Claes says:

    Impressive.

    Have you given any thought to the problem that applications that once were run as local applications gets exposed over the network when run this way? Do you have any good idea how to avoid that security concern? I have often thought if not HTTP-over-domain-sockets is feasible for this kind of use. Or perhaps some kind of file-based authorization. Both requires support from the web browser though.

  7. Juergen Mangler says:

    @Claes: e.g. https & pkcs12 client (personal) certificates. Standard tech since ages. You could issue and revoke certificates for frieds or other people that are intended to use the app.

    @alexander: again, impressive. Does it cover everything, or are there missing pieces? Drag & drop between real apps and browser (firefox) gtk apps :-)? Wouldn’t it be better to have new windows in new tabs (e.g. the preferences in the evince example)?

  8. jayz says:

    OMG

    How awesome is that!

  9. nona says:

    Very cool.

    I suffer from a lack of imagination though – I can’t find a use for it off the top of my head. ssh -X or vnc would probably work better for me. Maybe if you lack a vnc client.

    Any way, just because I can’t come up with a use for it, doesn’t mean it’s not awesome.

  10. Is the “Broadway” name a reference to the old X consortium’s plans to put X apps on the web via a browser plugin?

  11. alexl says:

    Alexandre:

    I don’t quite see the point of using broadway for local stuff. You might as well use your native display technology (X11/wayland) and then use a web widget like GtkWebkit if you want to develop with that kind of APIs.

    J5:

    I’ve been recommended orbited before, but its just never gonna be fast enough, as it is polling. How often do you have to poll to get some reasonable rate of pointer motion events. Way to high to make any sense.

    Tiago:

    No, firefox has many direct X calls.

    Conrad:

    I expect to be doing window management on the client side in the html/js, so it will get a WM of sorts.

    Claes:

    Well, you obviously don’t want to run this on any server and allow anyone to access your apps, much as you don’t allow anyone to remotely connect to your X server. So, SSL, certificates and general network security needs to be applied.

    Juergen:

    Lots of stuff are missing, no dnd or c&p for instance.
    I’ve thought about actually making the windows real toplevel browser windows, but there is some complications with that wrt things like menus where we really need full control of window positioning and no window frames/titles.

    James:

    Yeah, it is. Glad someone else remembered that. :)

  12. Sandino Flores says:

    Ey! Impressive work.
    I tried in macosx, since the x11 backend is currently broken in mac I only enabled quartz and broadway.

    However, I can only see the UI rendered in Firefox 4, but I cannot interact with it.
    (WebSockets are enabled).

    Is the X11 backend mandatory to get bradway working?
    If not, what could be preventing it from working properly?

    Thanks.

  13. alexl says:

    Sandino:
    Weird, i don’t expect it to require anything from the X11 backend.
    Do you have a proxy enabled or something like that?

  14. Sandino Flores says:

    No, there is no proxy, but it could be that I’m too ignorant and Snow Leopard could do have a built-in firewall. I’ll try again. Thanks for the quick answer.

  15. Sandino Flores says:

    It works now! After I rebooted the mac, firefox 4 now accepts input.
    Somehow, although I had quitted firefox 4, it was not taking the changes.

    Time to experiment!

    Thanks again.

  16. Carla says:

    Funny enough I could not watch your screencast in my Mozilla browser, though I had no problem to watch it on Seamonkey

  17. me says:

    How much slower will it be if latency goes up?

  18. [...] this is a very exciting project and Larsson has made a video of broadway in [...]

  19. Wasif Hasan Baig says:

    Very Coooool…looking forward to it :)

    Once these web-apps (as I see them now :p) get a WM so that we can drag them in the browser space, is it possible to have multiple apps open in the same tab?

    It would be nice to have that functionality :)

    Really appreciate your work :)

  20. ohwow says:

    Oh man, this is amazing! Good luck with your coding!

  21. First off, I think all the efforts you have made are very cool. There is a significant amount of effort to pull this off. Hats off for that.
    Build the software and the mobile flat-fee unlimited-usage network will come :) The problem is mobile 3G/4G networks are not currently flat fee unlimited usage.

    Your comment about compiling gtk3 with the “–enable-x11-backend –enable-broadway-backend” says it all. Everything everywhere is dependent on X11 protocols being implemented on all the the devices running firefox in the first place. You’ve confirmed yet again X11′s networking window api is the still the coolest on the planet.

    Why do this? Why not compile and run X11 apps everywhere? Most devices these days have X11 built-in anyways. If I understand correctly there would be less overhead in terms of OS api calls and in terms of screen display data net bandwidth. Are you doing something innovative for the screen display data that cuts down on the bandwidth? Is the screen display data all svg-based? My point is the net wasn’t intended on sending receiving screen presentation data. It was intended on sending/receiving precious keyboard/other input devices’ data along with pseudo-database query/response data to both ends and then let the two ends worry about the presentation/gui. The rest of this GUI/GTK/X11 presentation/gui/data is expensive $$$ on bandwidth. Something tells me it wouldn’t work so well on 3G/4G networks because you would be passing too much data that costs you send and receive. Have you tried google maps in satellite display on a cell phone and looked at your 3G/4G phone bill? Everything on cell-phone 3G/4G is usage-based so you would notice it more quickly that perhaps you don’t want to use gtk apps in a web browser. Doing gtk in a firefox browser is like streaming movies because essentially that’s what it is and more. I think your work is very cool, but it’s essentially $$$ to use on 3G/4G. From what I understand, the point of web browsers is you want your gui to be platform-independant and run your web pages on every device. It’s highly efficient and feasible for browser to run text web pages. Web page serving becomes less feasible the more images/video you put on the web page, not because you the developer can’t do it, but because the mobile bandwidth for sucking up that kind of image/video data is currently too expensive for the greater majority of the world’s population.

    Why does everything need to run in the web browser? If you’re going to run remote fat clients, why not run a native fat clients?

    All this is adding yet another layer of obfuscation for the developer and more importantly causing a bigger territory for malware to drop in and say hello.

    Call me conservative, but I find the current X.org/ssh/tomcat/apache solutions adequate for wanting to control a remote machine if that’s you’re intent. If you’re intent however is to run a gtk-based version of quake on a remote pc with your web browser, why not just run quake on your local machine? What’s the added value of running quake on the remote machine?

    Cheers

  22. alexl says:

    David:
    The –enable-x11-backend is just to keep your normal X-using app working on your machine. Its fully possible to build gtk with only the broadway backend, and to use a non-gtk firefox (such as the win32 one) with broadway apps.

    I don’t expect broadway to be used everywhere. Its certainly not a great target for smartphones, but neither is X11.Which one will use the most bandwidth depends a lot on what the app draws, but broadway is not always worse than X11.

    However, I don’t expect this to replace typical web apps. Its more like a way to remote apps on a lan or suchlike, typically to systems that don’t have X11 installed (i.e. most non-linux systems).

  23. [...] Alexander Larsson, Ubuntusecrets.it, [...]

  24. Bill says:

    Could you use this to run WinXP apps on a Windows 7 machine? In particular, WinXP pps with embedded 16-bit com objects on a 64-bit Win 7 OS? If this can enable existing non-Win7 compatible apps to run on a the new OS, there is a huge market for it.

  25. Bill says:

    Corrected typing mistakes:

    Could you use this to run WinXP apps on a Windows 7 machine? In particular, WinXP apps with embedded 16-bit com objects on a 64-bit Win 7 OS? If this can enable existing non-Win7 compatible apps to run on the new OS, there is a huge market for it.

  26. Bender says:

    This is great stuff! It is like personal (sic!) cloud but I/You control it! I am using KDE right now but THIS makes me think of all the possibilities of remote work etc. Great!!

  27. ylorak says:

    This is very cool and truely shows the flexibility of a lot of the background technologies (x11, firefox, gtk etc).

    But besides being a nice tech demo, what is the added advantage over current remote desktop solutions?

  28. HidekiAI says:

    Neat! I’m tired of Qt and was thinking to switch to Gtk, so this will be my justified reasons to start my switch…

    As for usage, there are times when I want to demo something but I don’t want to create an account for them to “ssh -X” to my system. I can (hopefully) just open up port 8080 temporarily, have my router forward to my dev-box, and even if sluggish, they can try the demo (and critique on UI, etc) – as compare to rather than send the source and say “here, compile this…” (with a constant back-and-forth of “I can’t compile because it’s missing libXYZ.abc.so”). I’m sure it’s not as easy as it sounds, but well…

    I think it has potential… “OKTHXBYE!”

  29. Natanael L says:

    My first thought when reading this was of Mozilla’s latest project Chromeless. Have you taken a look at that? It aims to create a platform for developers that let them use HTML5 and sych web technologies to create apps that looks and feels like any other native app.
    It would be great with a Chromeless based app that would act as a specialized client for Broadway’ed GTK apps, making it as seamless as it can get.

  30. [...] GNOME и мантейнер таких проектов, как Nautilus, Gnome-vfs и Dia, сообщил о том, что GDK-бэкенд (GIMP Drawing Kit), позволяющий [...]

  31. Mark says:

    @ylorak
    You don’t need a client (like VNC,SSH,etc). Just a web browser, and that is pretty much standard everywhere.

  32. Zarquon says:

    Gnome.org’s mime type is incorrectly set to text/plain for the WebM file. It should be video/webm.

  33. Jack says:

    I guess this is one area where Qt will be falling short, soon enough.

    From what I’ve read, it seems like it shouldn’t be as taxing as sending a constant stream of X11 from host to client, especially since you’re compressing the data and polling for image diffs only when necessary. I would expect applications like word processors/music players/any useful program without a ton of imagery to perform better with this than X11 over SSH. Also, there is an awesome benefit here- theoretically, any platform with HTML5 could run these applications. Not every platform that runs HTML5 can run X11 or accept its output. VNC isn’t a great general-purpose option for running single applications.

    Also, any platform that can currently compile and run GTK and HTML5, but not X11, could run Gnome applications locally. As far as I know, the lack of X11 and Mesa on Android phones is the biggest reason why you can’t run Linux applications written in the regular toolkits.

    Couldn’t I, theoretically, compile and run GTK+ 3 on Android with the proper dependencies and display the applications in a browser? As far as I’m aware you can install most of what GTK needs natively on Android. Correct me if I’m wrong.

    In that case, it would be practical for smartphones, although I expect there to be considerable overhead. Then again, how much of the memory/processing needed to run Gnome applications is attached to performance in X itself? Maybe broadway could perform better than X in some cases?

    Even though I love Qt for development, I might take a chance with GTK+ if this is in any way feasible. I’d really like to hear more about the details and what’s necessary to wrap up the remainder of the work with the non-supported bits. XD I imagine it must be annoying explaining this in the face of multiple walls of text- your feedback is quite appreciated.

  34. Miguel says:

    I’m sure this could be the biggest jump gnome do i last 10 years.
    Thanks for your work!

  35. [...] GNOME и мантейнер таких проектов, как Nautilus, Gnome-vfs и Dia, сообщил о том, что GDK-бэкенд (GIMP Drawing Kit), позволяющий [...]

  36. [...] 赶紧前往作者博客观看视频演示吧! [...]

  37. [...] lo cuentan los chicos de WebUpd8, donde hablan de Gtk+ 3.2 enlazando a un artículo de Alexander Larsson en el que este desarrollador de Red Hat habla de la posibilidad de contar con un backend gtk para [...]

  38. Diego says:

    Why do you guys spend time in useless stuff like this? Why not spend time in something more important like Wayland instead?

    Seriously.

  39. ghurley says:

    @Tiago

    Yes you can. Just type chrome://browser/content/browser.xul
    into the address bar and press Enter.

  40. crncosta says:

    Congratulations!

    Did you tried to run a VTE based terminal (like GNOME Terminal) inside the browser? I would like to know if it works, specially Vim or Emacs! :-)

    Anyway, Good job… thanks for your effort.

  41. Caglar Oral says:

    I totally agree with Diego. They want to be part of ChromeOs thats the reason behind it. Its totally useless why dont you spend time porting gdk to native framebuffer or wayland or cairo and get rid of lots of X11 dependencies?

  42. [...] wie, ale już działa. Twórcą kodu i autorem małej prezentacji jest Alexander Larsson, który pisze na swoim blogu o postępach [...]

  43. Dominic Iadicicco says:

    This is the future of computing, Gnome and all applications. Great work work to whoever is working on this project. :)

  44. [...] Want to see something wild? Operating system that runs GTK+ 3.2 (currently found in GNOME 3.0) can have applications launch in a web browser, thanks to HTML5 gdk “broadway” backend. A demo of this miracle is attached below. What [...]