Broadway adventures in Gtk4

One of my long running side projects is a Gtk backend called “Broadway”. Instead of rendering to the screen this backend creates a HTTP server that you can connect to, and then exposes the UI remotely in the browser.

The original version of broadway was essentially streaming image frames, although there were various ways to optimize what got sent. This matches pretty well with how Gtk 3 rendering works, particularly on Wayland. Every frame it calls out to all widgets, letting them draw on top of a buffer and then sends the final frame to the compositor. Broadway just inserts some image delta computation and JavaScript magic in the middle of this.

Enter Gtk 4, breaking everything!

However, time moves on, and the current development branch of Gtk (which will be Gtk 4) has completely changed how rendering works, with the goal of doing efficient rendering on modern GPUs.

In the new model widgets don’t directly render to a buffer. Instead they build up a model of how the final result should look in terms of something called render nodes. These describe rendering as a tree of highlevel operations. The backend (we have software, OpenGL and Vulkan backends) then knows how to take this description and submit it to the GPU in an efficient way. This is somewhat similar to the firefox WebRender project.

Its would be possible to implement the broadway backend by hooking up the software renderer, letting it generate a buffer and then send that to the browser.  However, that is pretty lame!

CSS comes to the rescue!

Instead I’ve been looking at making the browser actually draw the render nodes. Gtk defines a lot of its UI in terms of CSS these days, and that means that the render nodes actually are very close to the CSS rendering model. For example, the basic drawing operation are things like rounded boxes with borders, shadows, etc.

So, I was thinking, could we not take these render node and turn them into actual DOM nodes with CSS styles and send them to the browser. Then every frame we can just diff the DOM trees, sending the minimal changes necessary.

Sounds crazy right? But, it turns out to work pretty well.

Check out this example page which I created with the magic of “save as”. In particular, try zooming into that page in the browser, and play with the developer tools inspector to see the nodes. Here is a part of it zoomed in:

The icons and the text are not CSS, so they don’t scale, but look at those gorgeous borders, shadows and gradients!

Entering the 3rd dimension!

Particularly interesting is the support in Gtk for general 3D transforms. This maps well to the CSS transform on the browser style.

Check out this example of a spinning-cube transition. If you open up the browser inspector you can see that each individual element in the cube is still a regular CSS box.

Some technical notes

If you look at the examples above they all use data: uris for images. This is a custom mode that lets “screenshots” like the above work. Normally broadway uses blobs for the images.

Also, looking at the examples they seem very heavy in terms of images, as all the text are images. However, in a typical frame most of the render tree is identical to the previous frame, meaning any label that was used in the last frame need not be sent again. In fact, even if it changes position in the tree due to a parent node changing (scrolling, cube-switching, etc) it can still be reused as-is.

However, text is clearly the weak point in here. Unfortunately HTML/CSS has no low-level text rendering APIs we could use. I’m considering generating a texture atlas with pre-rendered glyphs that can be reused (like CSS sprites) when rendering text, that would mean we will have to download less data at least. If anyone has other ideas I would love to hear about it.

12 thoughts on “Broadway adventures in Gtk4”

  1. this is so cool 🙂

    regarding text rendering: what about shipping a WASM version of some text renderer, and then just sending text. The client could then render text and alt-text itself.

    Reply

  2. “Unfortunately HTML/CSS has no low-level text rendering APIs we could use.”
    TLDR
    Why Not .woff ?
    Web Open Font Format (WOFF)

    “The icons and the text are not CSS, so they don’t scale”
    icons = SVG
    text = woff
    ———————————————————–
    And does the font really matter anyway? (Metric-compatible)
    And don’t forget a11y (Accessibility)
    Text should be Machine Readable, Copy Paste-able and Vector Zoom-able.
    https://www.google.com/search?q=.woff
    https://www.w3.org/Fonts/WOFF-FAQ
    https://developer.mozilla.org/en-US/docs/Web/Guide/WOFF
    https://en.wikipedia.org/wiki/Web_Open_Font_Format
    https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face
    https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Fonts
    https://www.w3schools.com/css/css_font.asp

    https://www.google.com/search?q=metrically+compatible
    https://en.wikipedia.org/wiki/Liberation_fonts
    https://en.wikipedia.org/wiki/Croscore_fonts

    https://wiki.archlinux.org/index.php/Metric-compatible_fonts
    Arial = Liberation Sans = Arimo
    Times New Roman = Liberation Serif= Tinos

    Also With “Cascading Style Sheets” (CSS) You can Specify Multiple Metric-compatible fonts and they will Cascade

    on a User Script I am working on I use
    “color: #0000EE; color: -moz-hyperlinktext; color: -webkit-link;”
    Will First set the color to 0000EE
    Then Override it with -moz-hyperlinktext if you are on Firefox
    and then override it again with -webkit-link if you are on Chrome
    So only the Last Valid Style is used.

    I go a little over board and use both
    [scriptexternallink]:link { color: #0000EE; color: -moz-hyperlinktext; color: -webkit-link; }
    [scriptexternallink]:visited { color: #551A8B !important; color: -moz-visitedhyperlinktext !important; color: -webkit-link !important; }

    l[i].style = “color: #0000EE; color: -moz-hyperlinktext; color: -webkit-link;”;
    l[i].setAttribute(“scriptexternallink”, “3rdParty”);

    So in our example
    https://wiki.archlinux.org/index.php/Metric-compatible_fonts#List_of_metric-compatible_fonts
    Microsoft; CrOS; GNU; woff; woff2; Liberation;
    Times New Roman; Tinos; FreeSerif; LiberationSerif.woff; LiberationSerif.woff2; Liberation Serif;
    Put the Preferred Font last and then go backwards
    Worst/Bad case scenario the browser won’t choose a Metric-compatible font and the text will be slightly too big/small/long/short

    Thanks. I really hope Debian/Buntu/gNewSense/Trisquel/PureOS doesn’t Disable Broadway.
    https://manpages.debian.org/stretch/libgtk-3-bin/broadwayd.1.en.html
    http://manpages.ubuntu.com/manpages/bionic/man1/broadwayd.1.html
    (It looks like it is enabled now)
    http://www.tarnyko.net/en/?q=node/7
    http://www.tarnyko.net/dl/gtk.htm
    (3rd party Windows build Too Apparently)
    Maybe one day I will finally use Broadway as a remote desktop.
    I love how you are going from bitmap to vector. Bitmap never felt right in a browser. And if it saves bandwidth that is a huge bonus too.
    Now I can hopefully Save Screen shots to SVG!!!
    Please Keep Text as Text.
    I love what you are doing.
    Keep up the Good Work 😀
    woff woff
    E-Bod

    Reply

    1. TLDR
      My Posted Examples above and below aren’t tested and may not actuarially Work.
      Google Fonts has Cantarell but i don’t think/know if it has any Metric-compatible fallbacks
      Consider Pros And Cons of Browser input types and Submits
      Consider Pros And Cons of strip down simple DOM with Custom Attributes and A Separate CSS.css file
      You can Create or substitute Widgets with CSS defined globally (It doesn’t need to be specified in every element)
      SCSS to Web Browser CSS with cascading fallbacks
      Consider a11y
      Broadway, Electron, emscripten, and webassembly Synergies
      (SDL Qt and unity3d) still using canvas
      Client/Front-end/Browser (Interactive JS in Browser) + BroadwayServer/WebAssembly/Backend
      ————————————–
      PS

      When i said “So in our example”
      I added a Disclaimer “This is not formatted right just an example of Cascading fallbacks”
      that wasn’t real code
      the &lt &gt got removed
      @font-face
      Defines the font-family and src
      https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face#Font_MIME_Types
      font/ttf font/otf font/woff font/woff2

      https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Fonts#Basic_example
      https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face/font-family
      font-family: “Helvetica Neue”, “Arial”, sans-serif;
      font-family: “font family”;
      font-family: ‘another font family’;
      Defines the Fallbacks

      Remember to test the rendering in Chrome Firefox Android iOS Old And New.
      “Chrome for Android” Has a ? for font-family

      Also
      “color: -moz-nativehyperlinktext;”
      Can set the link color but only on Firefox

      https://wiki.gnome.org/Projects/GTK/Accessibility
      https://developer.gnome.org/accessibility-devel-guide/stable/gad-how-it-works.html.en

      https://en.wikipedia.org/wiki/Cantarell_(typeface)
      https://fonts.google.com/specimen/Cantarell?selection.family=Cantarell
      https://fortintam.com/blog/2011/04/13/on-cantarell/

      I don’t know if anything is Metric-compatible with Cantarell but if the Font and size are flexible It shouldn’t matter too badly if it falls back to something else (that is if the page dynamically changes). End users are already allowed to modify their CSS themes.

      https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/checkbox
      https://www.w3schools.com/tags/att_input_checked.asp
      https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input
      https://www.w3schools.com/tags/att_input_type.asp

      You could consider using Real Browser input types
      Alternatively Define Your own Attributes

      -img style=”position: absolute; left: 0px; top: 0px; width: 14px; height: 14px;” src=”data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA4AAAAOCAYAAAAfSC3RAAAABmJLR0QA/wD/AP+gvaeTAAAA7klEQVQokcWRvU4CQRSFv7tSESstSLCl3aVwduAp5BFMTJCKhFDQ0BhDR9RoQQPhGSAkPsRkLVxbW0iwsLGwYyiQZfiLodHT3XPzndwf+Ev5YeFWDoUCpVtAM3UAI4EK74EagLedWLzY4Yuv9CNIbWl465C+gdkwr8Ke05O80k8C1VWM3CU7+qpQEuxgWVvonx6nK59f3x3Elp38dhyZhnscz1e6K3CVBMO7hZxTP7xGpg5w5ID2YzIeZbJnGUD9eCerrnTjF5OM64IATCfj5w0YhF4cmcpigz3gJmyh/xaZaxf6TV5wri/Z8bL/0RyFYT1aKcsTZAAAAABJRU5ErkJggg==” width=”14″ height=”14″-

      -input type=”checkbox” checked –
      -img GtkWidgetType=”GtkCheckButton”-

      img[GtkWidgetType=”GtkCheckButton”] {src: “GtkCheckButton.png”; src: “GtkCheckButton.svg”; width: 14px; height: 14px;}
      (I Haven’t checked that this works but it is just an idea)
      (You could use class instead of (or in addition to) attribute but for userscripts I avoid conflicts)
      (Actually you could do exactly what you are doing now and add class and custom attributes for a11y and CSS UserScript UserStyle Mods to use latter Remember the Alt text for images)

      OK so the checkbox Does Zoom in nicely on Google Chrome but on Firefox Its stays the same size
      https://stackoverflow.com/questions/306924/checkbox-size-in-html-css
      When you force it to scale it pixelates on FF

      https://www.w3schools.com/howto/howto_css_custom_checkbox.asp
      This is all CSS an no SVG nor Bitmap.
      https://www.w3schools.com/howto/howto_css_custom_checkbox.asp
      https://www.w3schools.com/html/html_form_elements.asp
      https://www.w3schools.com/html/html_form_input_types.asp
      https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input

      So you Can use the “input” and Override it with your own CSS
      Or you can use img (input div button etc) and Define your own CSS without the Browser trying to interpret its an input

      https://www.w3schools.com/howto/howto_js_tabs.asp
      onclick=
      https://www.w3schools.com/howto/howto_js_accordion.asp
      DOM CSS and JS are Separate (DOM defines a class) (CSS uses Class) (JS loops through class and adds addEventListener)

      https://developer.mozilla.org/en-US/docs/Web/CSS/:hover
      hover doesn’t need JS

      http://www.w3schools.com Menus has a lot of examples of things that could be GTK widgets that move and are defined in CSS.

      https://en.wikipedia.org/wiki/Client-Side_Decoration
      https://wiki.gnome.org/Initiatives/CSD
      https://electronjs.org/
      https://www.npmjs.com/package/electron-gtk-theme
      https://github.com/jaszhix/electron-gtk-theme
      https://github.com/electron/electron/issues/1065
      https://github.com/electron/electron/issues?utf8=✓&q=GTK
      https://www.google.com/search?q=electron+gtk

      See what Electron is doing. maybe they already solved your problem.
      Maybe you can make Broadway, Electron, emscripten, and webassembly share some overlapping code styles parsers generator etc. (Electron is MIT License)

      https://developer.gnome.org/gtk3/stable/chap-css-overview.html
      https://developer.gnome.org/gtk3/stable/theming.html
      https://blog.gtk.org/tag/css/
      https://blog.gtk.org/2016/06/22/adwaita/
      http://www.cssdesk.com/rCLHq
      https://blog.gtk.org/2019/01/14/theme-changes-in-gtk-3/
      https://github.com/GNOME/gtk/tree/master/gtk/theme/Adwaita

      So if I understand this correctly GTK uses
      “HTML-like DOM structure, with nodes and classes.”
      and
      “SCSS”
      “Adwaita is a complex theme, so to keep it maintainable it’s written and
      processed in SASS. The generated CSS is then transformed into a gresource file
      during gtk build and used at runtime in a non-legible or editable form.”

      If you Can Get your hands on the “SASS” file You can change the SASS to CSS generator to add Browser Specif Fallbacks without mucking around in the clean SASS file.
      then GTK can take the “HTML-like DOM structure” and make it into Renderable HTML5/CSS
      The DOM can use either standard or Custom Attributes that the CSS.css styles
      Put the two Together and the user can override the CSS file if they want.
      So you Make “DOM nodes with CSS styles”

      “So, I was thinking, could we not take these render node and turn them into actual DOM nodes with CSS styles and send them to the browser. Then every frame we can just diff the DOM trees, sending the minimal changes necessary.”

      Then the Question Arises If you want to use Real Browser input types and Submits or Define Your own Attributes
      Ether way you can strip the CSS and formatting out of the “DOM trees” and have the CSS in it’s own File.
      CSS in the DOM tree can override the CSS file.

      Maybe you can have the inputs and drop down menus and dynamic elements on the Client Browser and submit the inputs back.
      (Rather than have a Static DOM tree that is constantly updating)
      Is it possible to have a normal DOM Interactive web page (Client/Front-end/Browser) that just communicates with the BroadwayServer/WebAssembly/Backend
      Some interactive elements need their own JS but others don’t need js
      Broadway can inject the CSS and JS into the DOM
      but alternatively it can keep the DOM clean and have the CSS in in css.css and the JS in a js.js
      the DOM would need Class and/or Attributes
      Adwaita can have it’s own CSS.css and JS.js
      the Broadway Front end can handle standard GTK interacting with the user
      And the Broadway/WebAssembly Server/Backend can handle the backend stuff
      Does the WebBrowser really need to wait for updates (New DOM Frames) from the backed/server for some of the user interactions? Can some of this be done Browser side JS? GTK3 Broadway Canvas/bitmaps has no choice but to get new frames, but you aren’t using canvas.

      How does Pure DOM + CSS (No JS) & Constant DOM Frame updates compare with
      DOM + CSS + JS & Rare DOM Frame updates
      If the JS has to loop through every element by Class/Attributive that could slow down every time there is a new Frame but it could eliminate the need for new dom frames. (New DOM frames can even get their own Window/Tab/Frame) Data could be piped in and out and synergize with WebAsembly.

      (HTML5 removed support for some frame features.)
      https://www.w3schools.com/tags/tag_frame.asp
      “HTML Tag. Not Supported in HTML5.”
      https://www.w3schools.com/tags/tag_iframe.asp
      align frameborder longdesc marginheight marginwidth scrolling “Not supported in HTML5.”
      sandbox srcdoc (New in HTML5)

      https://en.wikipedia.org/wiki/Front_and_back_ends
      https://en.wikipedia.org/wiki/Client%E2%80%93server_model

      https://www.google.com/search?q=xul+to+html

      Get a look at the Competition
      https://blog.qt.io/blog/2018/05/22/qt-for-webassembly/
      https://wiki.qt.io/Qt_for_WebAssembly
      https://www.google.com/search?q=Qt+WebAssembly
      https://blog.qt.io/blog/2018/05/22/qt-for-webassembly-examples/
      http://example.qt.io/qt-webassembly/widgets/richtext/textedit/textedit.html
      It looks like webassembly + canvas (No DOM)

      https://www.google.com/search?q=emscripten+sdl
      https://github.com/emscripten-core/emscripten/wiki/Porting-Examples-and-Demos

      dime a dozen “WebAssembly native variant of Broadway” idea
      https://www.spinics.net/lists/gtk/msg22158.html
      https://www.spinics.net/lists/gtk/msg22159.html
      https://mail.gnome.org/archives/gtk-list/2017-June/msg00023.html
      https://mail.gnome.org/archives/gtk-list/2017-June/msg00024.html
      https://mail.gnome.org/archives/gtk-list/2017-June/thread.html#00024

      http://processingjs.org/
      https://github.com/processing-js/processing-js/
      Processing.js is Discontinued
      Processing.js seems much slower than Native Java and it was before asm.js and webassembly were a thing
      But it let you run Actual JAVA in the Browser.
      It makes it’s own
      script type=”text/processing”
      and renders everything in canvas

      So it is possible to convert non Javascript into JS/WASM or have a JS/WASM interpreter not js scripts

      Broadway, Electron, emscripten, and webassembly seem to all need to render GTK themes in the Web Browser
      webassembly (SDL Qt and unity3d) as far as i can tell right now just dumps bitmaps into canvas but maybe it doesn’t have to be that way.
      Electron I’m assuming uses REAL HTML and Javascript and CSS (but might use webassembly + canvas for parts that are games)
      Maybe Most of it (Or at least the GTK Widgets) can be real DOM + CSS and then only the parts that can’t be HTML DOM CSS JS WASM can be bitmaps in canvas.
      (So i’m assuming SDL Qt and unity3d all put everything as bitmap in canvas but these are “Games” that don’t translate into DOM+CSS)

      If you are using real DOM + CSS does the DOM even need to be updated (like canvas bitmaps) that often and can it change with user inputs directly in the browser. Can More of the Dynamic Rendering and CSS be done Client side in the Browser rather than Server side.
      (I think CSD is the opposite of true GTK + Broadway with Client side theming despite using the term “client side”)

      Broadway+Android (Instead of RDP) Broadway+WSL (Instead of Xserver) Broadway+SSL/TLS (Instead of RDP)
      Broadway instead of native GTK port.
      WebAssembly+GTK+PWA+Offline+Add-to-Home-Screen

      Is emscripten/webassembly + GTK a thing yet? and can it be accomplished in DOM instead of canvas?
      Can Broadway bring back the old server/client model that Xserver used to have and Wayland Got rid of?
      Can the GTK program run Completely in the Browser as an WASM Offline progressive web app with Add to home screen?

      You are the Expert This is only my 2 cents
      I’m sorry if i wasted too much of you time. I Hope Broadway doesn’t get killed off before we figure out how to use it (Mainstream Adoption).
      Vector DOM vs Bitmap Canvas sounds really interesting.
      Thank you for your time
      E-Bod

      Reply

  5. Warning!
    https://www.w3schools.com/about/about_copyright.asp
    w3schools is “Fair Use” “Favorable Use” “non-profit teaching” Only
    Kind of Defeats the whole point of posting code snippets in the first place and then saying “Fair Use” only.
    You can redact those links if that is a problem.
    (Or collapse remove shorten my comments if they are too long)

    https://www.w3.org/Consortium/Legal/2015/copyright-software-and-document

    https://developer.mozilla.org/en-US/docs/MDN/About#Copyrights_and_licenses
    Code samples and snippets MIT and/OR PD CC0

    Better to look at MDN examples

    All I meant to do was bark WOFF.

    Reply

  6. I was wondering, since GTK 4 architecture is now similar to what Webrender is doing, why didn’t you join force with Mozilla and adopted Webrender as the rendering engine for GTK ?

    Reply

    1. Similar doesn’t mean the same unfortunately. Gtk has a massive history, and switching out rendering like this isn’t as easy as it sounds. Its an interesting idea though.

      Reply

  7. Hello

    Within GTK3 + BROADWAY, some applications have a problem.
    For example:
    There are apps that do not find a desktop environment and when they do not, they do not run or they recycle an error.

    Has GTK4 + BROADWAY been fixed?

    Thanks

    Reply

  8. Hi! Do you have any information about webassembly support in the near future? QT has announced this week the webassembly support, and provided a few impressive online demo. I would like to stick with GTK, I feel it more intuitive for me, but from a technology perspective, webassembly support is something to concider.

    Reply

    1. I don’t personally have any plans for WebAssembly, and I don’t actually think it is very interesting either for Qt or Gtk. If your goal is to run something entirely in the browser, then it should be a native web design, anything else will just be too heavy and large.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *