Hi, I’m Gaurav Agrawal, a member of the GNOME Engagement Team. I recently had the chance to interview Alyssa Rosenzweig, who is a lead developer at Panfrost project which is a free and open source driver Mali Midgard and Bifrost GPUs. Alyssa spent her summer as an intern at Collabora working on improving Panfrost’s OpenGL ES 2.0 userspace, which helps GNOME Shell work fluidly on supported Mali Hardware.
How about we kick off with a little bit of background on Panfrost?
Panfrost is a free, open-source graphics stack for Arm Mali GPUs, focused on the popular Midgard series. While these chips are popular among Android devices, they have been historical thorns in Linux’s side, due to the closed nature of the official drivers. Panfrost aims to change that, bringing the benefits of open-source to the Mali world.
What started out as a small community reverse-engineering effort has now matured into a reliable OpenGL ES 2.0 driver. Since May, I’ve been using Panfrost as my daily driver to program Panfrost. And yes, I’m answering these questions from a machine with Panfrost!
How did you get involved with the Project’s team/founded the team.
I’m passionate about spreading free software across the entire stack. To me, it’s not enough to have a free kernel; we also need a free desktop environment like GNOME. Yet it’s not enough to have just a free kernel and free desktop environment — we need free drivers and free firmware. Researching the state of free firmware for x86 systems, I realized that for long-term success, free software needs to win on Arm platforms, where free firmware at the lowest levels is still an option on systems like the Rockchip RK3399. These Rockchip systems have gained considerable mainline support, including support for the on-board video processing unit, thanks to past Collabora contributions. The future looked bright for Linux on Arm.
Unfortunately, despite these freedom wins, these Arm boards featured Mali GPUs, whose proprietary drivers prevented free software from truly taking off here. Frustrated with the GPU serving as the sole obstacle to a modern fully open source laptop, two years ago I purchased a development machine with a Mali — and the rest is history.
We will love to know, what were the issues with existing proprietary Arm drivers, which users were facing?
The issue with proprietary drivers is both practical and philosophical, and the proprietary Mali drivers are no exception. 3D acceleration is a de facto requirement of the modern system; even if a user is not interested in video games, they still need acceleration to run software using OpenGL like GNOME with full performance. Thus, philosophically, the requirement of the proprietary drivers for OpenGL support prevents normal usage of systems with Mali with free software.
Practically, the proprietary drivers pose a number of challenges for Linux users. Arm’s userspace drivers require the use of Arm’s kernel drivers. While these kernel drivers are technically open source, they are tightly coupled with the proprietary stack, which prevents their integration with the upstream “mainline” kernel. Today, most users never have to think about installing a kernel; the kernel for their hardware is part of their distribution, and distributions can easily maintain support for any hardware supported upstream. But a Linux user that needs a Mali chip cannot rely on their distribution for the kernel; the driver is maintained out-of-tree and requires a complex porting process to work against a normal upstream kernel. Far too often, users will resort to use outdated, buggy, insecure, downstream kernels, simply because they cannot use the mainline kernel if they need graphics.
Panfrost changes that. Our kernel module is designed for open-source and is included in the mainline kernel. Likewise, our userspace implementation is open-source and part of the upstream Mesa project, shared with the open-source Linux drivers for Intel, AMD, and Broadcom GPUs. Thus, with Panfrost, Linux users can install the distribution of their choice, using a modern, secure upstream kernel, while 3D graphics works out of the box.
Your project focuses on improving Panfrost’s OpenGL ES 2.0 userspace, we will like to know what this is about, and how it will benefit others?
OpenGL ES 2.0 is the core API for graphics on Arm platforms. Although newer versions of OpenGL ES exist, most software a user will encounter day-to-day can run on OpenGL ES 2.0. By focusing on this API, Panfrost is able to provide a smooth user experience where it counts.
Panfrost uses the open source Mesa implementation of OpenGL ES 2.0 to provide this experience to users. Mesa provides the OpenGL frontend via the common open-source “Gallium” API. Panfrost is a Gallium driver, thus enabling OpenGL ES 2.0 apps to run atop Mali with no proprietary components.
But Panfrost goes further! OpenGL ES is the “embedded subset” of OpenGL, the API used more commonly on Linux. The proprietary userspace drivers only support OpenGL ES, with no support for desktop OpenGL, leaving Linux users forced to specially compile software or use fickle translation layers. Fortunately, Panfrost provides a solution!
Leveraging the power of a strong open-source community via Mesa and Gallium, Panfrost is able to support OpenGL 2.1, a “common denominator” API prevalent on Linux. Other drivers have contributed to the desktop OpenGL support in Mesa and Gallium, and via this shared open-source framework, this work is shared and everyone benefits — including Panfrost users.
In practice, this support means a user running a distribution like Debian can install desktops like GNOME and have acceleration work out of the box. Whereas the proprietary userspace would leave a would-be GNOME user to fend for herself, Panfrost provides a smooth, Linux-first experience.
We really want to know how are you so creative with commit messages 😉 (… , :^, ./test-clear works, woo, I think I got it ?, Fix textures \0/, 🤔 , I tried…, Hmm )
Programming is mentally draining for some and physically draining for others. For me, I think programming is _emotionally_ draining. By the time my code works, sometimes you just have to let out all that emotion into the nearest text box. Sometimes that’s IRC, and sometimes that’s the commit message 🙂
You went on a bug fixing adventure with GNOME, and we are excited to know what treasures you got 😉
Sometimes debugging feels like chasing my tail. But that’s not a problem — I’m not going in circles; I’m spiraling out and learning so much along the way. Sometimes that knowledge doesn’t help fix the bug, but it’s always a treasure!
GNOME offered no shortage of treasures. I installed a standard GNOME system from my distribution, which was built with OpenGL 2.1, rather than OpenGL ES 2.0. While OpenGL 2.1 has been tested with Panfrost, at the time, we had not tested it as extensively as ES 2.0, so there were all sorts of little gotcha’s I discovered. For instance, desktop OpenGL uses a slightly different texture specification mechanism, which challenged our previous texture implementation and demenaded a refactor — something I never would have noticed if I weren’t bringing up an app like GNOME.
_The_ bug, as it were, was unrelated to my research into complex topics like textures and tilers. No, in fact, it was a trivial piece of code related to the viewport descriptor. Panfrost’s implementation was correct for OpenGL ES 2.0, but again, OpenGL 2.1 offers more flexibility, so our implementation did not work there. After an agonizing bug search, a little bit of robustness improvements to the viewport code made all the difference in the world, and a moment later, I was running GNOME.
It will be really interesting to know how you all got nearer to the “Rasterization Discard” with the work “Scoreboard Implementation” on Mali’s Tiled Architecture, and we are curious to know simple explanation of these terms.
In graphics with OpenGL, the fundamental unit is a “draw”. Each draw has a pair of shaders, small programs running on the GPU. The first shader is the vertex shader, which determines where on the screen the GPU should render. The second shader is the fragment shader, which determines what colours the GPU should render. For an application like GNOME, these shaders are simple, copying the images of windows onto the screen. For a game, these shaders can be arbitrarily complex to implement fancy rendering algorithms.
Mali’s architecture subdivides draws into two parts, a vertex job (running a vertex shader) and a tiler job (setting up a fragment shader). If a simple app submits 100 draws, the driver will generate 100 vertex jobs and 100 tiler jobs. But these jobs have to run in a specific order: while all of the vertex jobs could run simultaneously, each tiler job (fragment shader) can only run once the corresponding vertex job (vertex shader) has run. Mali lets each job “depend” on other jobs, so one job can only run after its dependencies have run, much like a package in a package manager only installing once its dependencies were installed.
Originally, we hardcoded these dependencies, but this proved inflexible. This hardcoding was replaced by a high-level description of each job’s dependencies, so an automatic algorithm can compute in which order jobs need to be specified. This algorithm is knowing as “scoreboarding”.
The benefit of this automatic approach is seen in “rasterization discard”, an OpenGL feature that runs vertex shaders but does not draw anything to the screen. On Mali, that means we generate a vertex job, but we _don’t_ generate a tiler job. When we hardcoded jobs, this asymmetry was a problem, but once we implemented an automatic algorithm, this is as simple as just… not generating a tiler job. In negative lines of code, we can implement rasterization discard!
What are some popular devices that you believe can adopt your work, and through them it will benefit lots of people ?
A number of Arm-based Chromebooks use GPUs supported by Panfrost, including my personal development laptop, the Samsung Chromebook Plus. Collabora has contributed to the open-source mainline support for ChromeOS on these Chromebooks, and as a result of our open-first approach, Linux users of these Chromebooks benefit from a well-supported mainline stack. With Panfrost in the upstream kernel, these Chromebooks work on mainline _without_ sacrificing critical hardware support!
Beyond Chromebooks, based on the same high-performance RK3399 chipset, the Linux community’s own Pinebook Pro will support Panfrost. On a smaller scale, Mali chips are ubiquitous in phones and tablets; Panfrost will help the postmarketOS project achieve one of their stated goals, running the mainline kernel on phones for Linux for long-term support.
Looking back so far, what did you folks enjoyed the most with working around FOSS projects and communities?
The people! No matter where I go in the FOSS world, there’s always a friendly face. In real life, I can sometimes be timid, but online in the open source community, I can always hop into an IRC channel and strike up a chat with a developer or a fellow user. That sense of community, that despite coming from a myriad of countries, timezones, and identity backgrounds, we’re all united by a common purpose — that is a breath of fresh air from societies so focused on individual competition.
What are some inspirational lessons which you want to share with us, which will inspire newcomers contributing to FOSS ?
You can make a difference in the world of free software. It’s easy to be jaded and feel that nothing we do matters, that the tides of the world are set in stone at the whim of someone more powerful. Sometimes that can be true, but in the free software community, everyone has a chance to make a difference. If you can code, find an interesting open-source project to contribute to. If you’re multilingual, the community is always looking for translators. And even if you’re just an end-user, testing counts — if something doesn’t look right or doesn’t seem right, file a bug report and let the developers know! Or, if you’re a little extroverted and knowledgable on some software (even as a user!), try hanging out on your favourite project’s IRC channel and helping other users with the software — you never know whose day you could be improving with some patience and a little kindness. Little changes add up to making free software the beautiful place it is today, and you can help.
How can someone become involved?
Try Panfrost! Panfrost is shipping with Linux 5.2 and Mesa 19.2, arriving in popular distributions shortly. If you have a board with a compatible Mali GPU, grab the open-source stack and start testing! Maybe try your favourite desktop environment, or grab an open-source video game compatible with OpenGL 2.1, like SuperTuxKart. Give it a spin!
Edited for content and grammar. Images provided by Alyssa Rosenzweig, licensed CC-BY-SA 4.0.
That’s a really clear explanation of what vertex shaders and fragment shaders are. I don’t think i ever really understood it before 🙂
This is great. I hope panfrost is added to https://mesamatrix.net/ so I can watch another implementation of OpenGL slowly turn green.
It’s absolutely great to see this thorn of a proprietary driver finally being pulled out. Now I can setup my kiosk PCs anew and finally get native resolution without hassle. Thanks so much!