Yes, it was something to do with controlling terminals

Back in June, I wrote about the difficulty of killing the dbus-monitor subprocess when Bustle is recording the system bus:

But sending SIGKILL from the unprivileged Bustle process to the privileged dbus-monitor process has no effect. Weirdly, if I run pkexec dbus-monitor --system in a terminal, I can hit Ctrl+C and kill it just fine. (Is this something to do with controlling terminals? Please tell me if you know how I could make this work.)

Greetings, past me! I’m just like you, only 6 months older, and I’m here to tell you how to make this work.

When you run a process in a terminal (emulator), its controlling TTY is a pseudoterminal (hereafter PTY) slave1 device. When you press Ctrl+C, the terminal doesn’t send SIGINT to the running process; it writes the ^C character '\x03' into the corresponding PTY master. This is intercepted by the kernel’s TTY layer, which eats this character and sends SIGINT to the (foreground) process (group) attached to the PTY slave. It doesn’t matter if that process (group) is owned by another user: being able to write to its controlling terminal is the capability you need.

Armed with this knowledge, and some further reading on how to set the controlling TTY, it was reasonably easy to wire this up in Bustle:

You may wonder how the second step works in the Flatpak case, where the pkexec dbus-monitor process is not spawned directly by Bustle, but by the Flatpak session helper daemon. Well, we don’t get a chance to run our own code between fork() and exec(), but that’s okay because the session helper does exactly the same thing for us. (I think this is why Ctrl+C works in a host shell running within Flatpak:d GNOME Builder.)

  1. not my choice of terminology []

Bustle 0.7.1: jumping the ticket barrier

Bustle 0.7.1 is out now and supports monitoring the system bus, without requiring any prior system configuration. It also lets you monitor any other bus by providing its address, which I’ve already used to spy on ibus traffic.

Screenshot: Bustle monitoring IBus messages.

Bustle used to try to intercept all messages by adding one match rule per message type, with the eavesdrop=true flag set. This is how dbus-monitor also worked when Bustle was created. This works okay on the session bus, but for obvious reasons, a regular user being able to snoop on all messages on the system bus would be undesirable; even if you were root, you had to edit the policy to allow you to do this. So, the Bustle UI didn’t have any facility for monitoring the system bus; instead, the web page had some poorly-specified instructions suggesting you remove all restrictive policies from the system bus, reboot, do your monitoring, then reimpose the security policies. Not very stetic!

D-Bus 1.5.61 added a BecomeMonitor method explicitly designed for debugging tools: if the bus considers you privileged enough, it will send you a copy of every future message that passes through the bus until you disconnect. Privileged enough is technically implementation defined; the reference implementation is that if you are root, or have the same UID as the bus itself, you can become a monitor.

Three panels: woman smiling while looking at computer screen; mouse pointer pointing at 'Become a fan' button from an old version of Facebook; woman has been replaced by a desk fan.

Bustle, which runs as a regular user, needs to outsource the task of monitoring the system bus to some other privileged process. The normal way to do this kind of thing – as used by tools like sysprof – is to have a system service which runs as root and uses polkit to determine whether to allow the request. The D-Bus daemon itself is not about to talk to polkit (itself a D-Bus service), though, and when distributed with Flatpak it’s not possible for Bustle to install its own system service.

I decided to cheat and assume that pkexec – the polkit equivalent of sudo – and dbus-monitor are both installed on the host system. A few years ago, dbus-monitor grew the ability to dump out raw D-Bus messages in pcap format, which by pure coincidence is the on-disk format Bustle uses. So Bustle builds a command line as follows:

  • If running inside a Flatpak, append flatpak-spawn --host
  • If trying to monitor the system bus, pkexec
  • dbus-monitor --pcap [--session | --system | --address ADDRESS]

It launches this subprocess, then reads the stream of messages from its stdout. In the system bus case, polkit will prompt the user to authenticate and approve the action:

Screenshot: Authentication is needed to run /usr/bin/dbus-monitor as the super user.

Assuming you authenticate successfully, the messages will flow:

Screenshot: Bustle recording messages on the system bus.

There are a few more fiddly details:

  • If the dbus-monitor subprocess quits unexpectedly, we want to show an error; but if the user hits cancel on the polkit authentication dialog, we don’t. pkexec signals this with exit code 126. Now you know why I care about flatpak-spawn propagating exit codes correctly: without that, Bustle can’t readily distinguish these two cases.
  • Bustle’s old internal monitor code did an initial state dump of names and their owners when it connected to the bus. dbus-monitor doesn’t do this yet. For now, Bustle waits until it knows the monitor is running, then makes its own connection to the same bus and performs the same state dump, which will be captured by the monitor and end up in the same stream. This means that Bustle in Flatpak still needs access to the session and system buses.
  • Once started, dbus-monitor only stops when the bus exits, it tries and fails to write to its stdout pipe, or it is killed by a signal. But sending SIGKILL from the unprivileged Bustle process to the privileged dbus-monitor process has no effect. Weirdly, if I run pkexec dbus-monitor --system in a terminal, I can hit Ctrl+C and kill it just fine. (Is this something to do with controlling terminals? Please tell me if you know how I could make this work.) So instead we just close the pipe that dbus-monitor is writing to, and assume that at some point in the future it will try and fail to log a new message to it.2
  • Why bother with the pcap framing when Bustle immediately takes it back apart? It means messages are timestamped in the dbus-monitor process, so they’re marginally more likely to be accurate than they would be if the Bustle UI process (which might be doing some rendering) was adding the timestamps.

On the one hand, I’m pleased with this technique, because it allows me to implement a feature I’ve wanted Bustle to have since I started it in 2008(!). On the other hand, this isn’t going to cut it in the general case of a Flatpaked development tool like sysprof, which needs a system helper but can’t reasonably assume that it’s already installed on the host system. Of course there’s nothing to stop you doing something like flatpak-spawn --host pkexec flatpak run --command=my-system-helper com.example.MyApp ... I suppose…

  1. or version 0.18 of the spec if you’re into that []
  2. I’ve realised while writing this post that I can bring “some point in the future” forward by just connecting to the bus again. []

Bustle 0.5: Gtk+-3-ier, hidpi-friendlier

I finally replaced my vintage 2008 ThinkPad X200s, after months of agonising over which of keyboard, form factor, hidpi display, and software freedom to compromise on. Just in the nick of time, Dell released a developer edition of their widely-lauded XPS 13, which is spot on: same comfortable form factor (with 2015-era thin-ness); huge display with minimal bezel; conservative, usable keyboard layout; supplied and supported with free software; and the Sputnik team are very amiable on Twitter. I’m very happy with it.

I was less happy with how ludicrous Bustle looked on it, with its total ignorance of hidpi scaling and retro Gtk+ 2 stylings. I finally found the spare evening I mentioned 18 months ago and freshened it up to not let the amazing screen down.

Bustle 0.5.0

Source tarball and x86-64 binary available from the usual place. The freedesktop.org git repository is temporarily out-of-date due to some poorly-synchronised GPG and SSH key migrations, so for now it’s at GitHub. Update: the freedesktop.org git repository is current again!

(Hey, actual Bustle users! How do you feel about it as a piece of software? I’d be interested to hear.)

Bustle 0.4.3: I think you mean ‘fewer crashy’.

It’s Bustle release season! Here is version 0.4.3’s flagship new feature:

Bustle not crashing when it can't connect to the bus.

(Previously, it would crash.) I fixed a couple of other crashes, too, spurred by Sujith Sudhi reporting a i486-only crash. I feel compelled to point out that all of these crashes occurred in C code or at the inter-language boundary.

No Gtk+ 3 yet, I’m afraid, though experimental support in the Haskell binding was released a couple of days ago so it’s just a matter of a spare evening…

Meanwhile, why not follow my latest Twitter bot, @fewerror? It will provide you with 100% accurate corrections on a tricky point of English grammar.

Bustle 0.4.0: push button, receive D-Bus traffic

Breaking with the “every six months or so maybe” release tradition, here’s the second Bustle release of the month. What’s the new hotness this time? You can record D-Bus traffic by just clicking File → New, and watch the diagram being drawn before your very eyes. After years of on-off development, Bustle can finally liberate you from needing to open a terminal to monitor D-Bus traffic1. Here’s a super-brief video tour.

Bustle 0.4.0 screenshot

Grab it for x86_64 or i486 today! Unlike the previous release, these work on both Debian and Fedora and have a strong change of working on pretty much anything with a modern-ish Glib and Gtk+22. (Thanks to my fellow Collaboran Guillaume for testing these tarballs, and to Scott Tsai for his suggestion.) Source and so on at the usual location.

Today’s surprising Bustle-assisted observation is that switching to and from the Shell overview causes the Shell to retrieve /desktop/gnome/shell/windows/button_layout from GConf 29 times. (Presumably that number is proportional to the number of windows I have open?) This was extremely useful for testing the live-logging feature, but is perhaps not ideal in many other ways.

  1. “Occupy dbus-monitor” is arguably an unexciting slogan. []
  2. …inasmuch as any version of Gtk+2 can be called “modern”, that is. []

Bustle 0.3.1 provides 50% of your daily allowance of D-Bus message bodies

It’s a cold evening here in Cambridge, but I’m being kept warm at Collabora Towers, sipping a revitalizing mug of fresh applicative functor soup.

A mere five months after I demoed its features at the Desktop Summit in Berlin, here’s Bustle 0.3.1. Whereas previous versions of Bustle only recorded and showed you the names, senders and destinations of D-Bus messages, this version also records and shows you the contents of the messages.

Screenshot of Bustle 0.3.1

The statistics page also takes advantage of this new information: you can now get statistics about the sizes of messages in the log. Grab your copy today from the usual location. Beside the source, I’ve also uploaded a 64-bit binary tarball to save you some compilation time. Give me a shout if you have trouble with it. 32-bit version to follow when I get my chroots straightened out.

I have good news and bad news. Good news: here’s a 32-bit binary tarball. Bad news: seems like Debian and Fedora have differently-sonamed libpcaps. Why is distributing software so tedious?

Bustle 0.2.5: now with statistics

Bustle 0.2.5—“Why go all the way to Glastonbury to not watch U2 when you can just not turn on the BBC at any point this weekend to not watch them?”—is out now. This release adds a sidebar containing statistics about the log: namely, method call and signal emission frequency, and total/mean times spent in method calls.

Screenshot of Bustle 0.2.5

This code has mostly been sitting around in a branch since Plumbers in November. Sorry, dear users!

Bustle 0.2.4 released

Just released Bustle 0.2.4. Various bits of clean-up and bug fixes were kicking around in master that should have been released months ago. The viewer’s much more Postel-compliant, so if you’ve had trouble with your D-Bus logs being rejected with cryptic errors messages, you should upgrade. Also, Sergei Trofimovich contributed a build fix: thanks!

A mundane screenshot of Bustle 0.2.4

While we’re here: Bustle finally has a bug tracker! Bugs are in the Bustle component on freedesktop.org Bugzilla. And the Git repository has moved to freedesktop.org too. With luck, the next release will be more exciting. :)

Bustle 0.2.3: now with pairs of D-Bus traffic charts side-by-side

I just released Bustle 0.2.3. The notable improvement is that you can now view a pair of D-Bus traffic logs side-by-side on the same chart. So if you’ve taken a trace of the session bus and the system bus, and want to see how the bus traffic matches up on the two, this is the release you’ve been waiting for! (If not, well, I made the ugly pink lines a more tasteful grey, and fixed some bugs you never noticed.)

While I was refactoring to support the second log, I would have liked to have been able to run Bustle in a “batch” mode to render straight to a file, and then run some kind of visual diff tool to compare the output of the branch versus the last release. Coincidentally, when I opened my inbox, I found a mail requesting the same feature! I imagine that this could come in handy for producing automated reports: maybe you’d have a weekly cron job that produces some stats about the traffic using bustle-count and, if it goes significantly up or down, sends an email to tick off or congratulate the relevant team as appropriate with an attached diagram. ;-) If anyone fancied having a crack at this, it shouldn’t be too hard.

Bustle 0.2.1

A couple of days ago, I released version 0.2.1 of Bustle, someone’s favourite D-Bus profiler. As the version number suggests, there aren’t really any big new features; most of the changes just make it a bit nicer to use, like showing you all the bus names a service owns, ellipsizing strings, a slightly less spartan UI, etc. Having finally gotten around to cutting a release, I’ve started wondering what to work on next. There are various small things I have in mind, such as searching, filtering, integrating the various statistic tools (bustle-time and friends) into the UI, and so on, but it’d be nice to have a larger goal to work towards.

One recurring feature request is the ability to see messages’ arguments. This isn’t currently possible because the simple plain-text logs produced by the monitor (which is a variation on the theme of dbus-monitor --profile) only includes the message header. I’ve thought for a while that the right thing to do would be to log the raw dbus messages, together with a timestamp, but wasn’t sure what the files would look like. (Maybe shove the timestamps into the message headers?) Rob had a nice idea: why not log to pcap files? This avoids inventing a new format—the UI would just use libpcap and feed each message through the dbus parser—and would also let you look at the logs in WireShark, if you’re into that kind of thing. I’m hoping to find some time to give this a shot soon. (Maybe on a cold Christmas evening, in front of a fire?)

In the meantime, have a peek at what your D-Bus-using applications are up to, and let me know what’s missing!