Forward: For a drawn out post on next-generation X rendering, this blog entry is really short on eye candy. I apologize, but I’m at home, separated from my beloved eye candy, and figured I should write this while I felt motivated. As a way of forcing my own hand, I’m making a link now to a blog entry I haven’t yet written that will contain screenshots in the future 🙂

Next-Generation Rendering For the Free Desktop

For the past half year or so Red Hat’s desktop team has had people working toward making accelerated graphics rendering on the free desktop badass, but doing an ass job of actually talking about what they’re doing in a larger public / GNOME context. They’ve been doing a combination of experimentation (from that cracktastic OpenGL compositing/window manager luminocity to xsnow for the Xcomposite generation) and knuckle-down no-holds-barred infrastructure work (like making Win32 GTK work on Cairo so GTK can move to cairo as the default backend). With RHEL4 kicked out the door we’ve been able to rebalance day-to-day work on GTK and X onto other people to give the nextgenren hackers free hands. Currently the full-time nextgenren team at Red Hat is Owen Taylor (gtk/pango maintainer), Søren Sandmann (x hacker), Diana Fong (visual designer), Kristian Høgsberg (x hacker) and Carl Worth (cairo maintainer).

I’m really excited because these guy’s expertise is across a broad chunk of the rendering pipeline, from the toolkit down to the x server, which is going to give this effort the ability to work on this from a global perspective rather than optimizing the bits where we happen to have influence in. I’m doubly excited because other companies (well, Novell at least, but hopefully others will join) are starting to invest in this effort too!

I’m hoping to drag Owen into spinning this off into an umbrella effort (ala project utopia) to help maintain a coherent story/platform even as lots of people pour work into lots of different packages and distros. There are so many different ways to attack the X rendering issue that I’m a little worried about seeing a lot of fragmentation of effort and the result not being particularly coherent. I do hope people experiment with lots of different approaches, but I also really hope that in we can give developers a consistent platform for doing cool graphics on the free desktop. It would be a real shame to end up with the message in two years being “well, platform X has the feature you want, but you have to worry about also working with Y because X won’t work well on distro Z”. This sort of technology-choice morass can really dampen developers playing with this stuff and adding support all over GNOME, which is exactly the sort of quick-fiddling big-payoff stuff I think we’ll see a lot of as soon as this stuff starts landing. In other words, lets push toward the point where people can feel confident and start hacking up cool things for this system inside GNOME.

What It Might Look Like

A really good system needs to have lots of pieces in place all hooked together….its not something that can be hacked apart and replaced by arbitrary random incompatible bits (though there are points of commonality, such as OpenGL or Render). For example the pieces in one imaginable architecture – by no means the decided-upon final one or anything – might look like:

• A sophisticated drawing layer (cairo using glitz/opengl or render as backends)
• Stock renderers built on top of that drawing layer (pdf/ps rendering backed by cairo – such as Alex Larsson’s xpdf fork in evince, svg rendering backed by cairo, etc)
• A toolkit that agressively takes advantage of the features in the drawing layer, exposing them to applications and themes (gtk+)
• A window+compositing manager that can work closely with the toolkit but essentially takes the window contents as a static image in compositing (metacity with luminocity-like GL compositing manager features fused in to deal with window effects, synching up smooth resizing, live window thumbnailing, crazy pagers, etc)
• A hardware driver system to expose a low-level hardware accelerated rendering path to the drawing layer (opengl or render with hardware accel)

With that model we can implement things like:

• Toolkit themes that draw with layer blending effects, delightful bezier curves, and irritating alpha gradients
• Indiana Jones buttons that puff out smoothly animated clouds of smoke when you click on them
• Alpha transparency in applications whenever and wherever the urge strikes us
• Live window thumbnails
• Hardware accelerated PDF viewers
• Hundreds of spinning soft snowflakes floating over your screen…. without messing up nautilus
• A photograph of a field of long dry savanna grass as your desktop background… where the grass is gently swooshed around by a breeze created by moving your mouse across the background
• Windows that shrink scale and move all over the fucking place with cool animations
• Synchronized smooth resizing so there’s no disjunct between window borders moving and the contents redrawing (you should see the demos of this in luminocity… it really makes a difference in how real the interface feels, just as double-buffering did for stuff moving)
• A shared path between on-screen display and printing (using Cairo’s PDF/PS backends)
• Vector icons with very occasional super subtle animations rendered in realtime…a tiny fly which buzzes around the trash every several minutes, etc… think mood animations as in Riven (which as a total random aside is still a shockingly beautiful and atmospheric game years after it came out, postage stamp sized multimedia videos notwithstanding)
• Workspace switching effects so lavish they make Keynote jealous
• Brush stroke / Sumi-e, tiger striped, and other dynamically rendered themes where every button, every line looks a little different (need to post shots / explanation of this stuff, but another day)
• Progress bars made with tendrils of curves that smoothly twist and squirm like a bucket of snakes as the bar grows
• Text transformed and twisted beyond recognition in a manner both unseemly and cruel
• A 10% opaque giant floating head of tigert overlayed above all the windows and the desktop.
• etc etc. In short: awesome.

And that’s a conservative approach to this: each window essentially renders into a texture which are then combined in a separate rendering pass by the compositing manager. A lot of the work Diana does challenges our assumptions about what this rendering system should be able to do. For example, something as simple as a swoosh that cuts across both the window and the titlebar is currently very tricky. Diana’s work has illustrated something that may be obvious, but seems to be forgotten in the excitement to build the One True Graphics Pipeline (this does not exist!): Its very important to figure out many of the things you want to do with the graphics system before you get in too deep and dirty, because there are a lot of directions we could go that call for rather different architectural choices. To give one example, if we decided we really cared about having lots of animations throughout GNOME (this isn’t something we’re pushing, but we talked about it) that would dictate a very different approach from a graphics system where we really really cared about printing. You can’t always have your cake and eat it too… especially not when you consider implementation constraints.

Another example of how prioritizing “what do we want to improve with this” can change the direction: Since taking advantage of these new toys would require a new theme system, Havoc and I have been talking about how a very different theme / widget rendering system might work with this that allows for custom design of any window, widget, or anything in between. One of the things us designers have been experimenting with behind closed doors is what you can do with a window’s design when its not drawn out of a bunch of stock widgets but you have a freer hand. (This does not mean visual inconsistency, just as a magazine can maintain a consistent look but still do a fresh layout for each page using a mix of stock and new elements.) The results can be really good. No matter how good the artist, you can only get so far designing a crude palette of some fixed number of widgets which are then used in preset. A good theme/widget rendering framework would help us negotiate this balance between re-using stock elements, and overriding the rendering of widgets at appropriate points to customize how a “Control Center Preference Page” is drawn or to simply shift the text in buttons over 10 pixels to the left. Figuring out how this stuff works, or if we just want to leave the theming issue alone (which would sort of be a shame given how much of the old flooring we’re tearing up around it), may also have a significant impact on the final architecture.

A radical model (which also avoids multi-pass rendering without opening up security issues present in sharing direct access to existing graphic cards between processes) might involve a centrally rendered scene-graph where each client is given a subtree to add higher-level primitives. That could give us access to candy like pixel and vertex shaders (which we experimented with several months ago as part of rendering subtle but live backgrounds of grass fields, etc), which are attached to nodes on the render tree. Of course, there are many paths for leveraging shaders short of a full scene graph system. The scene graph model has a lot of significant concerns that are not as relevant to, say, 3D games where this model is common. Text rendering is one example.

Owen and company have slides from the X dev conf, but the punks did them as SVGs so unless you have their k-rad Cairo backed SVG slide presentation program, or if you’re willing to view slides in Inkscape… they’re not much good (though it is cool that you can find the slide you need using Nautilus thumbnails, but I digress) (hmmm, you can also open them in eog). Honestly, not the most inspiring OR detailed slides in the world either. I don’t think they’d had much sleep when they wrote them up. *grin*

Anyway… I’m rambling. I’ve given a couple points too much depth, most points not enough depth, many points I’ve missed, and doubtless some I’ve gotten wrong, but I knew if I waited to write the perfect post on this there’d be only more backlog of material to share… so a braindump it was. 🙂 I guess in the end I’m pretty excited. It feels like we’re running the last couple miles to get to the giant great-rendering payoff Keith Packard kicked off in the X world several years ago.

Code and stuff

• Cairo I think everyone knows about… writing for Cairo in Python or Mono is especially cool. Its really easy to get something that looks good going in short order. If you haven’t played with it, you should!
• Luminocity is in GNOME cvs with the module name ‘luminocity’
• Metacity compositing work is in ‘metacity’ with the branch ‘spiffifity‘
• GTK+ / Cairo integration…. gtk+ HEAD!

Apparently they also have a jhbuild setup that’ll build all this stuff thats headed for CVS in fairly short order.

And for my last point…

Hula!