We launched PipeWire last September with this blog entry. I thought it would be interesting for people to hear about the latest progress on what I believe is going to be a gigantic step forward for the Linux desktop. So I caught up with Pipewire creator Wim Taymans during DevConf 2018 in Brno where Wim is doing a talk about Pipewire and we discussed the current state of the code and Wim demonstrated a few of the things that PipeWire now can do.
Christian Schaller and Wim Taymans testing PipeWire with Cheese
Priority number 1: video handling
So as we said when we launched the top priority for PipeWire is to address our needs on the video side of multimedia. This is critical due to the more secure nature of Wayland, which makes the old methods for screen sharing not work anymore and the emergence of desktop containers in the form of Flatpak. Thus we need PipeWire to help us provide appliation and desktop developers with a new method for doing screen sharing and also to provide a secure way for applications inside a container to access audio and video devices on the system.
There are 3 major challenges PipeWire wants to solve for video. One is device sharing, meaning that multiple applications can share the same video hardware device, second it wants to be able to do so in a secure manner, ensuring your video streams are not highjacked by a rogue process and finally it wants to provide an efficient method for sharing of multimedia between applications, like for instance fullscreen capture from your compositor (like GNOME Shell) to your video conferencing application running in your browser like Google Hangouts, Blue Jeans or Pexip.
So the first thing Wim showed me in action was the device sharing. We launched the GNOME photoboot application Cheese which gets PipeWire support for free thanks to the PipeWire GStreamer plugin. And this is an important thing to remember, thanks to so many Linux applications using GStreamer these days we don’t need to port each one of them to PipeWire, instead the PipeWire GStreamer plugin does the ‘porting’ for us. We then launched a gst-launch command line pipeline in a terminal. The result is two applications sharing the same webcam input without one of them blocking access for the other.
As you can see from the screenshot above it worked fine, and this was actually done on my Fedora Workstation 27 system and the only thing we had to do was to start the ‘pipewire’ process in a termal before starting Cheese and the gst-launch pipeline. GStreamer autoplugging took care of the rest. So feel free to try this out yourself if you are interested, but be aware that you will find bugs quickly if you try things like on the fly resolution changes or switching video devices. This is still tech preview level software in Fedora 27.
The plan is for Wim Taymans to sit down with the web browser maintainers at Red Hat early next week and see if we can make progress on supporting PipeWire in Firefox and Chrome, so that conferencing software like the ones mentioned above can start working fully under Wayland.
Since security was one of the drivers for the move to Wayland from X Windows we of course also put a lot of emphasis of not recreating the security holes of X in the compositor. So the way PipeWire now works is that if an application wants to do full screen capture it will check with the compositor through a dbus-api, or a portal in Flatpak and Wayland terminology, and only allows the permited application to do the screen capture, so the stream can’t be highjacked by a random rougue application or process on your computer. This also works from within a sandboxed setting like Flatpaks.
Jack Support
Another important goal of PipeWire was to bring all Linux audio and video together, which means PipeWire needed to be as good or better replacement for Jack for the Pro-Audio usecase. This is a tough usecase to satisfy so while getting the video part has been the top development priority Wim has also worked on verifying that the design allows for the low latency and control needed for Pro-Audio. To do this Wim has implemented the Jack protocol on top of PipeWire.
Carla, a Jack application running on top of PipeWire.
Through that work he has now verified that he is able to achieve the low latency needed for pro-audio with PipeWire and that he will be able to run Jack applications without changes on top of PipeWire. So above you see a screenshot of Carla, a Jack-based application running on top of PipeWire with no Jack server running on the system.
ALSA/Legacy applications
Another item Wim has written the first code for and verfied will work well is the Alsa emulation. The goal of this piece of code is to allow applications using the ALSA userspace API to output to Pipewire without needing special porting or application developer effort. At Red Hat we have many customers with older bespoke applications using this API so it has been of special interest for us to ensure this works just as well as the native ALSA output. It is also worth nothing that Pipewire also does mixing so that sound being routed through ALSA will get seamlessly mixed with audio coming through the Jack layer.
Bluetooth support
The last item Wim has spent some time on since last September is working on making sure Bluetooth output works and he demonstrated this to me while we where talking together during DevConf. The Pipewire bluetooth module plugs directly into the Bluez Bluetooth framework, meaning that things like the GNOME Bluetooth control panel just works with it without any porting work needed. And while the code is still quite young, Wim demonstrated pairing and playing music over bluetooth using it to me.
What about PulseAudio?
So as you probably noticed one thing we didn’t mention above is how to deal with PulseAudio applications. Handling this usecase is still on the todo list and the plan is to at least initially just keep PulseAudio running on the system outputing its sound through PipeWire. That said we are a bit unsure how many appliations would actually be using this path because as mentioned above all GStreamer applications for instance would be PipeWire native automatically through the PipeWire GStreamer plugins. And for legacy applications the PipeWire ALSA layer would replace the current PulseAudio ALSA layer as the default ALSA output, meaning that the only applications left are those outputing to PulseAudio directly themselves. The plan would also be to keep the PulseAudio ALSA device around so if people want to use things like the PulseAudio networked audio functionality they can choose the PA ALSA device manually to be able to keep doing so.
Over time the goal would of course be to not have to keep the PulseAudio daemon around, but dropping it completely is likely to be a multiyear process with current plans, so it is kinda like XWayland on top of Wayland.
Summary
So you might read this and think, hey if all this work we are almost done right? Well unfortunately no, the components mentioned here are good enough for us to verify the design and features, but they still need a lot of maturing and testing before they will be in a state where we can consider switching Fedora Workstation over to using them by default. So there are many warts that needs to be cleaned up still, but a lot of things have become a lot more tangible now than when we last spoke about PipeWire in September. The video handling we hope to enable in Fedora Workstation 28 as mentioned, while the other pieces we will work towards enabling in later releases as the components mature.
Of course the more people interesting in joining the PipeWire community to help us out, the quicker we can mature these different pieces. So if you are interested please join us in #pipewire on irc.freenode.net or just clone the code of github and start hacking. You find the details for irc and git here.
Good job guys. I wondered if the day would ever arrive when JACK becomes fully integrated to a wider desktop AV system. I currently use KX-Studio’s jack-aloop daemnon to route all audio (inc PA), with QjackCtl at the helm. Your project could be the icing on the cake.
Nice! How complete is JACK support?
Did you try to run more complex JACK applications? in particular ones that rely on JACK’s latency API and jack-transport?
If I understand correctly, pipewire pushes sync downstream (like gstreamer), so I cannot see how latency compensation would work.
Is there a design-doc that outlines how audio is aligned? ie. How long has it been since audio on a given port arrived on the soundcard’s input; how long will it be until audio will arrive on the souncard’s output?
It also seems there are two context switches: to jack-client, back to pipewire, to next jack-client (rather than jack’s client-to-client wakeup), and potentially format conversions (rather than zero-copy shared buffers).
Are the plans to change this to behave like actual jackd?
I will ask Wim tomorrow to explain the Jack support here as I don’t know the implementation details, it is zero copy though from what I remember. From what I understand the Jack support is feature complete, but the code is still quite fragile and thus Wim or someone else needs to spend time going over it and clean it up and ensure it is robust and clean.
I don’t really get the necessity of removing pulseaudio. Its stable, featureful, and works on a large range of devices.
Is pulseaudio effectively abandonned, or, who besides you considers it legacy software already?
What is the opinion of pulseaudio developpers?
Is there a path to migrate most features of pulseaudio back into pipewire?
Your presentation is naturally biased towards pipewire but I really dont get the objective underlying issues, especially if pulseaudio is still maintained.
Thanks
We have good contact with the PA devs and as far as I know they like the design of PipeWire. The main two reasons to look at retiring PulseAudio over the long run is that it would be quite painful to build a security framework like we got in PipeWire into it, the other thing is that it would be nice to have a single system handling all 3 usecases. So once PipeWire is there it would be nice to eventually retire PulseAudio just to avoid maintaining an extra codebase for the consumer audio case. But time will tell of course and the question of features in PulseAudio that we are not sure we want to replicate in PipeWire, like the network streaming stuff, might mean PulseAudio will stay around for the long term or at least until Wim caves and implements the features in PipeWire.
Hi! I’m wondering, will it cover the artistic use case, something in the direction of Syphon? http://syphon.v002.info/
The idea is to be able to send video across programs. This is often done in Apple computers to generate computer graphics in one program (Processing, OpenFrameworks, VVVV, Jitter…) and then sending that video feed to a a different program (VJ software, projection mapping, further manipulation or recording…)
The answer is yes on all accounts
Hello
Gnu Linux really needs improvements in the management of video and music, but have some skill in programming it’s not enough to understand the daily uses and needs of users. You must also have a minimum of personal experience in the field, qualifications and certainly share your pipewire work with qualified users in these domains. So what are your qualifications and who are the people with whom you share your work to get the best for pipewire. So that gnulinux finally be in the first place in audio and video for everyone.
Thx
This all sounds awesome. Finally we’re getting usable audio under Linux! (That may be unfair, but the jack/pulse situation is a huge mess. Things that just work on Windows and macOS takes forever to set up on Linux, if you can get it working at all. I know people who migrated off Linux over this.)
BTW, can pipewire video be used for real-time game capture with OBS Studio etc?
@sqf: please stop promoting this false story that “things that just work on Windows and MacOS…”.
It is true that there are some problems on Linux, particularly due to the poor integration between JACK and PulseAudio that many distributions have caused via a combination of mistakes and deliberate choices.
But it is absolutely NOT true that all this stuff “just works” on Windows and MacOS. In particular, if you’re actually using JACK for the purpose it was intended for (routing audio and MIDI between applications), then you cannot do this at all on those platforms without some other infrastructure in place (such as JACK).
In addition, if music creation/pro-audio is important to you then don’t start by using a Linux distribution that is intentionally not designed for this (which is to say: most of them). In the Windows world, there has long been a tradition of “audio PCs” – special systems set up specifically for music creation/pro-audio workflows. Linux is no different in this respect.
Macs are bit different, but even there you can hear the rumblings from pro users that Apple are not paying enough attention to their needs anymore (they are, after all, a very small niche market).
It’s fine to complain about specific issues with Linux audio, but these handwaving complaints don’t serve anyone using Linux well. Please remember that JACK exists for a specific kind of workflow – it is not intended to be a part of a typical desktop’s users setup.
@Sqf
Jack Audio connection is an excellent sound server for music production. I also use it daily with audio, video software like mpv or mpd but also with firefox.
Without being perfect, we can say that the software tends to be user friendly and its design is close to the needs of the users and it is not a coincidence because qualified people from Computer Music Research Laboratory participate in its development.
In the other side we have pulseaudio which is actually a patch of the old wrinkled Alsa which does not solve absolutely any problem but only adds a new layer of complexity. A software written by a person who has never set foot in the world of audio, without any musical qualification and who has no idea of the uses and needs of end users.
With my experience of these last fifteen years between the world of developers of free softwares and the world of users, I can say what is the huge mess. It’s the ego present among each developers who claim one after the other to write the best and most beautiful, and who continually with his blinders, and still in 2018, forget at least primordial things: DESIGN, USERS, USERS DAILY NEEDS, USER FRIENDLY.
You can write the most beautiful code in the world, your software will go no further than the current niche without at least these primordial things.
cordially
Looks very promising. Thanks so much for doing this! I wish for quick completion of the remaining features and fast migration to pipewire. Our linux-based recording studios will shine ;)
I often work with Red Hat and have to join Bluejeans meetings from my Fedora laptop. Its a PITA to switch to X11 from Wayland constantly. Similarly we use Zoom internally and hangouts socially most days and the lack of screensharing is downright frustrating.
Looking forward to the day it works seemlessly!
Hello,
How about talking to the guys from Ardour and exchanging some valuable experience? And then sharing with us the details :)
https://community.ardour.org/node/15510
As someone who uses PulseAudio’s network streaming daily, I’m glad it’s at least being acknowledged, even if there’s no plans to support it immediately in PipeWire. I’d be rather saddened to lose it…
(Inversely, the idea of sharing video devices across low-latency networks might be interesting.)
[WORDPRESS HASHCASH] The poster sent us ‘0 which is not a hashcash value.