Profiling Rust Applications With Sysprof

Sysprof is an enormously helpful tool that can be used for identifying performance problems in applications. However, there are a few things that need to be considered in order to get meaningful and useful results, especially for Rust applications.

With this blog post I want to provide a short step-by-step guide on what requirements need to be met, so that hopefully more people can make use of Sysprof.

For reference, I am using Fedora Silverblue 39, and have GNOME Builder 45 installed as a Flatpak from Flathub. Sysprof must be installed as a regular (rpm) package on the host system:

On regular Fedora Workstation:

As an example for profiling I will use my application Fragments. The process and the results may differ on other distributions / systems, as this depends heavily on whether frame-pointers are enabled or not, for example. The latter is a hard requirement for achieving useful results with Sysprof.

Debug symbols

I start Fragments with Builder and select the “Run with Profiler” option.

Then I can open and inspect the generated syscap file with Sysprof:


I select fragments from the left sidebar so that I can see all the associated descendants. However, the results are not really helpful yet, as instead of the method names only “In File… org.gnome.Sdk….” is shown.

This is because the required debug symbols are missing and have to be installed:

Make sure that you install the correct .Debug runtime for the corresponding version, which you have also specified in your Flatpak manifest (runtime / runtime-version, in my case master).

Rust and frame-pointers

With the right debug symbols installed, the results look much more readable:

However, it is noticeable that apparently almost exclusively function names of libraries (Gio, Glib, Gtk, …) appear, and nothing of Fragments itself.

This is because Fragments itself must also be compiled with frame-pointers enabled, which we can enforce with the Rust flag force-frame-pointers=yes – so let’s do that.

With objdump we can check if the compiled binary has frame-pointers enabled (and the specific registers aren’t clobbered):

If we have done everything correctly, this command should not return any output. But we do get output, and see some functions from the Rust standard library, even though we have frame-pointers enabled… How come?

The reason for this is that Rust toolchain, and thus also the standard library, have been compiled without frame-pointers enabled (the Rust SDKs from Flathub are repackaging the official Rust provided binaries).

To fix this, we can use -Zbuild-std --target=x86_64-unknown-linux-gnu . This will recompile the standard library during the build process (like a regular crate), while respecting our set compiler options/flags, especially force-frame-pointers=yes.

Since build-std is currently unstable, we have to switch from rust-stable to rust-nightly. Once build-std gets stabilised, or the Rust toolchain is compiled with frame-pointers enabled, we can switch back to rust-stable again.

If we run the same objdump command from above again, we get no output anymore. Which means, hooray, our binary has now been compiled without frame-pointers getting clobbered / optimised away!

Now, for the first time, we are getting results with Sysprof from which we can derive useful information. For example, we see that the Client::torrent_files function takes quite a bit of time, which is because we have to deserialise the JSON we get from transmission-daemon, which can be quite time consuming with long responses (and unfortunately happens synchronously, and not asynchronously).

Bonus: Better function names

We can improve the display of the function names by setting the Rust flag symbol-mangling-version=v0.

This way we use the new Rust v0 mangling scheme instead of the (currently still default) legacy one. This has the advantage for us that the function names can be displayed better/more detailed in Sysprof, since Sysprof has direct support for the Rust v0 scheme and therefore does not have to fall back to the generic c++ demangler.

Before (legacy):

serde_path_to_error::de::deserialize

After (v0):

serde_path_to_error::de::deserialize::<&mut serde_json::de::Deserializer<serde_json::read::StrRead>, transmission_client::rpc::response::RpcResponse<transmission_client::torrent::TorrentFilesList>>

Conclusion

All the changes I have described in this blog post are included in this merge request: Fragments!168

For more detailed information on how to use Sysprof itself, I recommend the Sysprof GNOME Wiki page, which contains some usage examples.

Thanks to Christian for Sysprof, and for the ability to use Sysprof together with Flatpak, which is a great benefit, especially for image-based systems like Silverblue.

The road to Fragments 2.0

Things have been a little quiet around Fragments lately, but behind the scenes work has continued nonetheless.

Now, after a little over a year of work, I am proud to announce Fragments 2.0!

Technical difficulties

Until recently Fragments was written in Vala, and used libtransmission as a backend. This had some disadvantages, e.g. a vapi file has to be maintained (the glue between the Transmission C API, and the Fragments Vala code).

In addition to that there are considerations from the Transmission maintainers to replace the C API with C++. This would mean that Fragments would not be able to use the Transmission API without creating (and maintaining) another wrapper (C++ -> C). In the long run I didn’t think this was very realistic.

And to be honest, Fragments was my last Vala application, and I was thinking for a long time how I could replace it with Rust 😉

The new beginning

The first thing I needed was a suitable BitTorrent backend. Since I have neither the knowledge nor the time to write my own BitTorrent implementation in the long run, my only option was to fall back on something already existing. Unfortunately, there is currently no BitTorrent implementation written in Rust that is comparable to libtransmission in terms of stability and features.

After a few brainstorming sessions I came up with the idea to continue using Transmission, but not via the C API, but to use transmission-daemon, and then communicate via its HTTP RPC API. This has several advantages:

    • Clean abstraction between the BitTorrent backend, and the actual application (this has some nice side effects, more on that later!)
    • Already existing features in Fragments 1 can be ported directly
    • The later migration from Vala to Rust is completely seamless from the end-user’s point of view, as transmission-daemon can directly take over the already existing configuration files etc. from Fragments 1.

The implementation

I needed to create a new client that would allow me to communicate with the Transmission RPC API. This was fortunately not a huge challenge in Rust thanks to serde-json. To keep this abstraction consistent, I decided not to develop this client in-tree within the Fragments source code, but to publish it as a separate crate on crates.io: transmission-client

This crate takes care of RPC communication only, and can therefore be easily reused in other Rust apps. However, from a Fragments perspective, it would be beneficial if I would be able to use the Transmission API using the GObject type system. So I decided to develop a second crate: transmission-gobject. This wraps the API of the first crate into gtk-rs gobjects. This has the advantage that I can easily embed the transmission data in the Fragments interface, because everything is available as a object with properties, and models. I could therefore concentrate on the pure application in Fragments and didn’t have to worry about technical aspects of the BitTorrent backend.

To visualize the whole thing:

When starting Fragments, the transmission-daemon process also gets started automatically in the background.

Fragments 2.0

Due to the strict abstraction on the network level, there is a big advantage: It does not matter if Fragments connects to its “own” transmission-daemon, or if it connects to a remote transmission server. Likewise, Fragments can be used to remotely control another Fragments session.

For example, I have Transmission installed on my Synology NAS, and use it as a “download server”. I can now connect to it directly from within Fragments:

It is also possible to switch between the individual “connections” directly within the app:

Of course there are a lot of other new features, like a new torrent dialog window:

Torrent rows have context menus now to access important actions more quickly:

Session statistics:

The magnet link of already added torrents can now be copied to the clipboard:

Completely overhauled preferences with more options like setting a separate path for incomplete torrents, queue management, behavior when new torrents are added and much more granular network configuration with the possibility to test the connection:

Fragments 2.0 can be downloaded and installed from Flathub:

Thanks

I would like to thank Maximiliano and Chris, both helped me a lot with the Rust + GTK4 port, and Tobias who helped me with the updated design!

Introducing “This Week in GNOME”

I have been following the “This Week in Matrix” blog series with great interest for some time now, and wondered: Why isn’t there something like this for GNOME?
To summarize the principle in a few words: A short, weekly summary in which maintainers briefly announce what they worked on for the past week.

For example, the following may be included in a weekly summary:

    • General news about a project
    • Presentation of new projects
    • New features
    • Instructions / Tutorials
    • Conferences / Meetings
    • General interesting thoughts that might be of public interest
    • … and much more! Just scroll through the Matrix blog and you”ll understand the principle very quickly.

After discussing the idea with other GNOME members, and agreeing that this is an idea with great potential, I started to implement the necessary technical requirements. We ran it as an experiment with a reduced set of maintainers. Here is our very first issue!

This Week in GNOME: #1 Scrolling in the Dark

Read through the blog post – it’s full of exciting news, and that’s just the beginning!

How does it work?

A user sends a message in the TWIG matrix room, mentioning the bot at the beginning of the message:

The bot will automatically recognize this message, and save it. In order for this message to appear in the next summary, it must be approved by an editor. This is done by adding the “⭕️” emoji (only editors have this permission).

Likewise, editors can add messages to a specific section, or to a specific project.

In this example I have done the following

    • ⭕️: I have approved this message.
    • 📻️: I have added the project description “Shortwave” to this message
    • 🟢: I have added this message to the “Circle Apps” section.
When a week has passed, an editor will create a new summary: this is a list of all the pieces people have been reporting since the last summary. To issue it, an editors runs the “!render-file” command in the administration room.

All collected messages with related information will be summarized in a markdown document. This can be used to create a new blog post using Hugo for example.

The message shown above would result in the following (raw markdown preview using Apostrophe):

The technical basis for this is hebbot – a matrix bot I developed in Rust using the matrix-rust-sdk. I tried to make this bot as generic and adaptable as possible, so that other communities can reuse it.

There have already been failed attempts to do monthly summaries, so why should it work with a weekly rhythm?

There are several reasons why it is very difficult to publish a monthly summary blog in the long term:
    • The time period is too long. A lot happens in one month. The longer the period, the more difficult (and time-consuming!) it is to summarize what has happened. Do you remember what you did in detail during this month? No? Neither do I.
    • Someone had to prepare the information so it could be shared in the form of a blog post. Either a central editor does this, or the submitter does it themselves. Either way, it’s a tedious and time-consuming process that many people don’t want to do.

TWIG has the following advantages here:

    • It’s super easy and quick to share news. You just need to open your chat client and send a short message to the TWIG room. You just finished a new feature on your project? Send a short (!) message about it, so that it will appear in the next weekly summary. A few words and maybe a screenshot/video are totally sufficient, no need to write a detailed blog post! Fire and forget!
    • The administrative workload is very low. An editor only has to approve and categorize the messages, the bot does the rest completely automatically.

Let’s show the world together what we do!

I’ve been involved in the GNOME project for quite some time now, and can say from personal experience that an outsider has absolutely no idea how much valuable work is being done behind the scenes.

    • Give the community the opportunity to share information with a large mass. GNOME Foundation members have access to the WordPress infrastructure, but there are many members who are not part of the Foundation. For TWIG, in principle, information can be shared by anyone, no matter who, as long as it is relevant to GNOME and newsworthy.
    • News first hand. We all know what happens when news / information gets out to the public via 5 different detours. Most of the time important information is lost or distorted. With TWIG there is a reliable and central source of truth.
    • Attract interested people / newcomers. The more people become aware of something / see what is happening, the more interest there will be.
Let us know what you’re working on, what cool feature you have released, or what bugs you have fixed! Join #thisweek:gnome.org and drop a message, we’ll do the rest!

New Shortwave release

Ten months later, after 14.330 added and 8.634 deleted lines, Shortwave 2.0 is available! It sports new features, and comes with the well known improvements, and bugfixes as always.

Optimised user interface

The user interface got ported from GTK3 to GTK4. During this process, many elements were improved or recreated from scratch. For example the station detail dialog window got completely overhauled:

New station details dialog

Huge thanks to Maximiliano, who did the initial port to GTK4!

Adaptive interface – taken to the next level

New mini player window mode

Shortwave has always been designed to handle any screen size from the beginning. In version 2.0 we have been able to improve this even further. There is now a compact mini player for desktop screens. This still offers access to the most important functions in a tiny window.

Other noteworthy changes

    • New desktop notifications to notify you of new songs.
    • Improved keyboard navigation in the user interface.
    • Inhibit sleep/hibernate mode during audio playback.

Download

Shortwave is available to download from Flathub:

Shortwave – First stable release

Today, after nearly two years of development I’m very proud to say: The first stable version of Shortwave is now available! I have put a lot of time and effort into this project, now it is finally time to make it available for everyone :-).

What is Shortwave?

Shortwave is an internet radio player that provides access to a station database with over 25,000 stations.

Automatic recording of songs

When a station is being played, everything gets automatically recorded in the background. You hear a song you like? No problem, you can save the song afterwards and play it with your favorite music player. Songs are automatically detected based on the stream metadata.

Streaming

It’s possible to stream the audio playback to a network device, which implements the Google Cast protocol (e.g. Chromecast). So you can easily listen to your favorite stations e.g. from a TV.

Adaptive interface

The interface of Shortwave is completely adaptive and adapts to all screen sizes. So you can use it on the desktop, but also on your Linux (not Android!) based smartphone.

Access to a huge database

Shortwave uses the internet service radio-browser.info as station database. It contains more than 25,000 stations. This ensures that you will find every radio station, whether a known or an exotic one.

System Integration

Shortwave integrates into the GNOME Shell, by providing a MPRIS applet and a proper PulseAudio implementation.

Gradio???

… is definitely dead now. But don’t worry, you can migrate the data easily to Shortwave.

Gradio: Application Menu -> “Library” -> “Export” -> “Gradio Database Format”

Shortwave: “Import stations from Gradio”

Download

Shortwave is already available to download from Flathub!

 

Or install it with:

flatpak install flathub de.haeckerfelix.Shortwave

Have fun with it! And many thanks to all who supported me during the development. Especially the fabulous GNOME Podcasts team 🙂

 

Second Shortwave Beta

Today I can finally announce the second Shortwave Beta release! I planned to release it earlier, but unfortunately the last few weeks were a bit busy for me

Changelog

    • Improved startup time. Shortwave should start now way faster. (#399)
    • Song recorder path is now configurable through gsettings (#406)
    • Don’t record newsticker messages / songs which are too short (#372)
    • The google cast feature isn’t experimental anymore (#368)
    • Fix AAC streams playback (#404)
    • New button in the application menu for creating new stations
    • The GNOME Shell MPRIS applet now displays the station cover
    • And of course, a bunch of other improvements / bug fixes!

Other changes:

    • Upgraded the gtk-rs stack to 0.8.0
    • Gtk.Application and Gtk.ApplicationWindow is now subclassed
    • Shortwave uses now the new radio-browser.info API (#418)

You can download the latest beta release from Flathub (Beta)! Please test it extensively and report all issues!

flatpak install https://flathub.org/beta-repo/appstream/de.haeckerfelix.Shortwave.flatpakref

Or just click here to install it. You can find the project page here.

First Shortwave Beta

Earlier this year I announced Shortwave, the successor of Gradio. Now, almost 11 months later, I’m proud to announce the first public beta of Shortwave.

Shortwave is an internet radio player that lets you search for stations, listen to them and record songs automatically.

Automatic recording of songs

When a station is being played, everything gets automatically recorded in the background. You hear a song you like? No problem, you can save the song afterwards and play it with your favorite music player. Songs are automatically detected based on the stream metadata.

Adaptive interface

The interface of Shortwave is completely adaptive and adapts to all screen sizes. So you can use it on the desktop, but also on your Linux (not Android!) based smartphone. For this Shortwave uses the awesome libhandy library!

Streaming

It’s possible to stream the audio playback to a network device, which implements the Google Cast protocol (e.g. Chromecast). So you can easily listen to your favorite stations e.g. from a TV.

Access to a huge database

Shortwave uses the internet service radio-browser.info as station database. It contains more than 25,000 stations. This ensures that you will find every radio station, whether a known or an exotic one. And if something is really missing, you can add your station here.

Where can I get it?

This is the first beta version of Shortwave. All basic features should work, but issues can appear. If somethings is wrong, please open a issue report here!

You can get it from Flathub (Beta). Install it with

flatpak install https://flathub.org/beta-repo/appstream/de.haeckerfelix.Shortwave.flatpakref

or just click here.