A new accessibility architecture for modern free desktops

My name is Matt Campbell, and I’m delighted to announce that I’m joining the GNOME accessibility team to develop a new accessibility architecture. After providing some brief background information on myself, I’ll describe what’s wrong with the current Linux desktop accessibility architecture, including a design flaw that has plagued assistive technology developers and users on multiple platforms, including GNOME, for decades. Then I’ll describe how two of the three current browser engines have solved this problem in their internal accessibility implementations, and discuss my proposal to extend this solution to a next-generation accessibility architecture for GNOME and other free desktops.

Introducing myself

While I’m new to the GNOME development community, I’m no stranger to accessibility. I’m visually impaired myself, and I’ve been working on accessibility in one form or another for more than 20 years. Among other things:

• I contributed to the community of blind Linux users from 1999 through 2001. I modified the ZipSlack mini-distro to include the Speakup console screen reader, developed the trplayer command-line front-end for RealPlayer, and helped several new users get started.
• In 2003 to 2004, I developed a talking browser based on the Mozilla Gecko engine; it ran on both Windows and Linux.
• Starting in 2004, I developed a Windows screen reader, called System Access, for Serotek (which has since been acquired by my current company, Pneuma Solutions).
• I later worked on the Windows accessibility team at Microsoft, where I contributed to the Narrator screen reader and the UI Automation API, from mid 2017 to late 2020. (Rest assured, the non-compete clause in my employment agreement with Microsoft expired long ago.)
• For the past two years, I have also been the lead developer of AccessKit, a cross-platform accessibility abstraction for GUI toolkits. My upcoming work on the GNOME accessibility architecture will build on the work I’ve been doing on AccessKit.

The problems we need to solve

The free desktop ecosystem has changed dramatically since the original GNOME accessibility team, led by Sun Microsystems, designed the original Assistive Technology Service Provider Interface (AT-SPI) in the early 2000s. Back then, per-application security sandboxing was, at best, a research project. X11 was the only free windowing system in widespread use, and it was taken for granted that each application would both know and control the position of each of its windows in global screen coordinates. Obviously, with the rise of Flatpak and Wayland, all of these things have changed, and the GNOME accessibility stack must adapt.

But in my opinion, AT-SPI also has the same fatal flaw as most other accessibility APIs, going back to the 1990s. The first programmatic accessibility API, Microsoft Active Accessibility (MSAA), was introduced in 1997. Sun implemented the Java Access Bridge (JAB) for Windows not long after. What MSAA, the JAB, and AT-SPI all have in common is that their performance is severely limited by the latency of multiple inter-process communication (IPC) round trips. The more recent UI Automation API (introduced in 2005) mitigated this problem somewhat with a caching system, as did AT-SPI, but that has never been a complete solution, especially when assistive technologies need to traverse text documents. For decades, those of us who have developed Windows screen readers have been so determined to work around this IPC bottleneck that we’ve relied, to varying degrees, on the ability to inject some of our code into application processes, so we can more efficiently fetch the information we need and do our own IPC. Needless to say, this approach has grave drawbacks for security and robustness, and it’s not an option on any platform other than Windows. We need a better solution.

The solution: Push-based accessibility

We can find such a solution in the internal accessibility architecture of some modern browsers, particularly Chromium and (more recently) Firefox. As you may know, these browsers have a multi-process architecture, where a sandboxed process, known as the content process or renderer process, renders web content and executes JavaScript. These sandboxed processes have all of the information needed to produce an accessibility tree, but for various reasons, it’s still optimal or even necessary to implement the platform accessibility APIs in the main, unsandboxed browser process. So, to prevent the multi-process architecture from further degrading browser performance for assistive technology users, these browsers internally implement what I’ll call a push architecture. There’s still IPC happening internally, but the renderer process initially pushes a complete serialized snapshot of an accessibility tree, followed by incremental tree updates. The platform accessibility API implementations in the main browser process can then respond immediately to queries using a local copy of the accessibility tree. This is in contrast with the pull-based platform accessibility APIs, where assistive technologies or other clients pull information about one node at a time, sometimes incurring IPC round trips for one property at a time. In terms of latency, the push-based approach is far more efficient. You can learn more in Jamie Teh’s blog post about Firefox’s Cache the World project.

Ever since I learned about Chromium’s internal accessibility architecture more than a decade ago, I have believed that assistive technologies would be more robust and responsive if the push-based approach were applied across the platform accessibility stack, all the way from the application to the assistive technology. If you’re a screen reader user, you have likely noticed that if an application becomes unresponsive, you can’t find out anything about what is currently in the application window, while, in a modern, composited windowing system, a sighted user can still see the last rendered frame. A push-based approach would ensure that the latest snapshot of the accessibility tree is likewise always available, in its entirety, to be queried by the assistive technology in any way that the AT developer wants to. And because an AT would have access to a local copy of the accessibility tree, it can quickly perform complex traversals of that tree, without having to go back and forth with the application to gather information. This is especially useful when implementing advanced commands for navigating web pages and other complex documents.

What’s not changing

Before I go further, I want to reassure readers that many of the fundamentals of accessibility are not changing with this new architecture. An accessible UI is still defined by a tree of nodes, each of which has a role, a bounding rectangle, and other properties. Many of these properties, such as name, value, and the various state flags, will be familiar to anyone who has already worked with AT-SPI, the GTK 4 accessibility API, or the legacy ATK. It’s true that we’ll have to add several new properties, especially for text nodes. However, I believe that most of the work that application and toolkit developers have already done to implement accessibility will still be applicable.

Risks and benefits

I’ve written a more detailed proposal for this new architecture, including a discussion of various risks. The risk that concerns me most at this point is that I’m not yet entirely sure how we’ll handle large, complex documents in non-web applications such as LibreOffice. I specifically mention non-web applications here because web applications, such as the various online office suites, are already limited in this respect by the performance of the browser itself, including the internal push-based accessibility implementations of Chromium and Firefox. My guess is that, with this new architecture, applications such as LibreOffice will need to present a virtualized accessible view of the document, similar to what web applications are already doing. We’ll need to make sure that we implement this without giving up features that assistive technology users have come to expect, particularly when it comes to efficiently navigating large documents. But I believe this is feasible.

I’d like to close with a couple of exciting possibilities that my proposal would enable, which I believe make it worth the risks. Let’s start with accessible screenshots. Anyone who uses a screen reader knows how common, and how frustrating, it is to come across screenshots with no useful alternate text (alt text). Even when an author has made an effort to provide useful alt text, it’s still just a flat string. Imagine how much more useful it would be to have access to the full content and structure of the information in the screenshot, as if you were accessing the application that the screenshot came from (assuming the app itself was accessible). With a push-based accessibility architecture, a screenshot tool can quickly grab a full snapshot of the accessibility tree from the window being captured. From there, I don’t think it would be too difficult to propose a way of including a serialized accessibility tree in the screenshot image file. Getting such a thing standardized might be more difficult, but the push architecture would at least eliminate a major technical barrier to accessible screenshots.

Taking this idea a step further, because the proposed push architecture includes incremental tree updates as well as full snapshots, it would also become feasible to implement accessibility in streaming screen-sharing applications. This obviously includes one-on-one remote desktop use cases; imagine extending VNC or RDP with the ability to push accessibility tree updates. But what’s more exciting to me is the potential to add accessibility in one-to-many use cases, such as the screen-sharing features found in online meeting spaces. And while this too might be difficult to standardize, one could even imagine including accessibility information in standard video container formats such as MP4 or WebM, making visual information accessible in everything from conference talks to product demo videos and online courses.

Conclusion

Too often, free desktop platforms struggle to keep up with their proprietary counterparts. That’s not a criticism; it’s just a fact that free desktop projects don’t have the resources of the proprietary giants. But here, we have an opportunity to leap ahead, by betting on a different accessibility architecture. The push approach has already been proven by two of the three major browser engines, and while there are risks in extending this approach outside of the browser context, I strongly believe that the potential benefits are too big to ignore.

Over the next year we will be experimenting with this new approach in a prototype, collect feedback from various stakeholders across the ecosystem, and hopefully build a new, stronger foundation for state-of-the-art accessibility in the free desktop world. If you have questions or ideas don’t hesitate to reach out or leave a comment!