Julian's Code Corner

Projects that receive funding from NLnet are required to have their code audited for potential security issues. Ours was performed by Radically Open Security, a Non-Profit Computer Security Consultancy from the Netherlands.
Since Fractal, by design, doesn’t include much security critical code the security researcher extended the quick scan somewhat also to the matrix-rust-sdk.

I have been in direct contact with the security researcher and they kept me up-to-date about their findings. This way, I could already during the audit start to fix identified security issues. Luckily, no major security issue was identified.

The issues found were addressed by us in the following way:

• 4.1CLN-013 — Fractal client stores images containing malware on filesystem

This is mainly a problem of the matrix server that it doesn’t sanitize images. Images downloaded from the server are stored in the encrypted store. This was initially an issue but was resolved. Videos on the other hand are currently downloaded and stored in the cache unencrypted because of this issue.

• 4.2CLN-012 — Fractal’s markdown implementation hides URLs to possible malicious websites

To address this we now show the full URL when the user hovers a link in the room history. This was introduced in this merge request.

• 4.3CLN-011 — Fractal allows opening of .html and .htm files

This is a problem with any file downloaded from an untrusted source. The researchers suggested adding a warning dialog to ask if the user is sure they want to open the file. I don’t think adding a warning is sufficient to prevent users from opening files containing malicious code, especially since users often don’t read things and just click continue or end up confused. Also we recommend using Fractal inside a Flatpak sandbox, that uses a portal that asks with which application to use to open the file.

Additionally, we decided to remove the open file button from the room history to make sure that user can’t open them easily by mistake in this merge request.

• 4.4CLN-010 — Matrix server does not sanitize uploaded images

The matrix server should address this and we can’t really do anything about it locally.

• 4.5CLN-009 — Images are stored on disk unencrypted

Now all data is stored encrypted. See issue for more details.

• 4.6CLN-008 — Security impact not sufficiently documented

We documented this in our README in this merge request.

• 4.7CLN-007 — Sensitive data can be extracted from database

Now all data is stored encrypted. See issue for more details.

• 4.8CLN-006 — Fractal client supports weak TLS cipher suites

This would be something nice to have, unfortunately currently not possible. See this issue for more details.

• 4.9CLN-005 — Fractal client is able to connect with insecure TLS versions

See issue 4.9CLN-006.

 

You can read the full report of the security audit here.

Last July I have been to Berlin for the second Mini GUADEC Berlin.

It was great seeing many other GNOME contributors in person after such long time. The conference took place in an awesome location at the c-base.

During the event I worked on an offline indicator for Fractal and improving the error handling.

I also spent some time on the Flatpak CI, adding information on how to get aarch64 builds. I previously added automatic builds of the CI images for aarch64 of the CI images.

Niels De Graef and I had a discussion on how Contacts should use the tel:// URI to open calls apps. The spec for the URI requires it to be a globally unique identifier. In the case of Contacts we don’t always have a country code, thus the phone-number can’t be globally unique. In my opinion we should use the phone number as entered by the user and let the calls app decide if it can dial a phone call. For example, Android behaves like this.

Lastly, I looked at building tarballs automatically in CI after tagging a new release. The tarballs are especially useful for Flathub, because it builds apps in an environment without internet connection.

I would like to thank the GNOME Foundation for sponsoring my travel to Berlin and Sonny and Tobias for organizing the event.

It’s been a while since I last blogged about the state of Fractal-next. Even though I’m not great at writing updates we’ve been making steady and significant progress on the code front. Before we dive into our progress I want to say thank you to all contributors, and especially Kévin Commaille. Keep up the good work ❤️.

Encryption 🎉️

Fractal-next now has support for sending and receiving room messages and allows people to verify their newly logged-in session and other users via cross-signing.

Thanks to the matrix-rust-sdk we got room encryption, especially the key management relatively easily. Nevertheless, we had to to integrate the SDK into Fractal and figure out a user experience that would work well with our design patterns and integrate with the GNOME desktop.

I spent quite a bit of time with Tobias, our designer, figuring out a design for session and user verification. Most of that has landed now, and though it still needs more polish, it work pretty well already.

One of the bigger differences between Element and Fractal-next is that we require users to always verify their new session, because making encryption “optional” on login means you end up with a confusing, half-functional session.

To fully support Matrix E2E encryption we will still need to implement the key backup system (which is not yet supported by matrix-rust-sdk). Currently we also don’t have a way to turn on encryption for a room. And of course we also still need lot more testing to make sure everything works.

Session management

Session/device managment has been in Fractal-next for a while now, allowing people to log out of other sessions and check if other sessions are verified.

 

Many new features

Fractal-next is currently in a bit of an inbetween state, where we don’t quite have all features from the old GTK3 version yet, but at the same time we have some very advanced new features we never managed to implement there.

A good example of this are reactions added by Kévin. Also, we’ve had multi-account support since last summer, but still no support for logging out 😅️.

There are also so many other things being implemented in record time that I’m not always sure what we have already added and what not (Thankfully we have an issue tracker 😀️).
To mention a few recent additions:

• Moving rooms between categories via drag&drop as well as via a context menu.
• The most important part of the room details
• Switching between versions of rooms after upgrades
• Media viewer for media in rooms
• Rich replies

Updates about the NLnet Grant

If you remember my work on Fractal-next was sponsored by NLnet. The initial grant was to add end-to-end encryption to Fractal, which then lead to the rewrite and Fractal-next. Luckily they accepted to sponsor more work. Without their support it wouldn’t have been possible for me to spend so much time on this project.

I still need to finish some things up before I can wrap up the grant project, which I really hope to do it within the next few weeks. I have also been talking to NLnet to get another extension to keep working on Fractal-next full-time a bit longer, to get everything in shape for a first release later this year.

The following items are part of the grant project and still missing:

• Finish the message store/cache for the SDK:
  This was much more complicated then anticipated, since Matrix doesn’t supported a persistent local copy of a room’s timeline, because the Event Graph can change due to federation.
• The grant foresees an Accessibility Audit and Security Audit:
  These wouldn’t have made sense so far since the app isn’t complete enough yet, so this is still outstanding.

At FOSDEM 2022

Don’t miss my talk about Fractal-next at FOSDEM 2022. It will be part of the matrix-rust-sdk talk.

As you probably know already we are in the process of rewriting Fractal. Since my previous blogpost about Fractal-Next a lot has happened. We are now at the point where the bare minimum is working, including login, the sidebar, and the room history. These things are not totally complete yet, but it’s already possible to run Fractal-Next, and read and send messages.
The previous post was kept for a general audience, but this one goes into more technical details of the foundation we put in place for Fractal-Next. In this post I will try to provide a more detailed look into the machinery, to give newcomers a general idea on how Fractal works. This is just an introduction and we are working on improving the docs for the entire project.

Fractal-Next uses GTK4 and therefore also glib’s GObjects. Every major part of Fractal is implemented as a subclass of a GObject or a GtkWidget. This being said, anything that doesn’t depend on GTK or glib will go, whenever possible, into the matrix-rust-sdk so that other projects can benefit from it. In return this will reduce the code maintained by the Fractal team and it keeps a clear separation between UI code and the backend code.
Fractal’s old code base doesn’t use subclassing nor composite templates, because when Fractal was started the Rust bindings for GTK didn’t have support for it yet. Today we can conveniently create subclassed objects with great support for composite templates to define the UI.

Subclassing and composite templates

We make heavy use of subclassing and composite templates. This allows us to create a GObject that contains the data and lets the object worry about maintaining consistency of the data with the Matrix homeserver. The object itself makes the appropriate calls to the SDK and then it exposes the information as GObject properties. This way, without going into too much detail, any GtkWidget can simply bind to the object’s property and it doesn’t have to worry about updating, requesting, and not even about the actual data it displays, because the binding keeps the widget and the object property in sync automatically.
We create GObject wrappers for many cases but not for everything. Matrix is built around sending events to logical places called Rooms. Generally, Matrix events can’t change their content and therefore we don’t need to make sure they are kept up-to-date, therefore we can use the rust struct directly instead of wrapping each single Matrix event into a GOjbect. It would mean that we need to writing a lot of additional code that for the definition and creation of the object. In short, it would result in a lot of overhead without any real benefit. Therefore we are using the Rust struct returned by the SDK directly to create the event widgets.

Async calls

You may wonder how we handle async functions from the matrix-rust-sdk. It’s simple, we created do_async() (thanks to slomo for the help making the function much nicer). The function takes two async closures, one that is executed on the tokio runtime and a second one that is spawned to the main context used by GTK. Obviously this function can only be called from the thread where the main context is running, but that’s not a problem for us since GTK can only run on a single thread and Fractal is mostly single threaded as well. We only use tokio with multiple threads for the SDK.

For example, the method to get the display name of a room looks like the following code snippet. self.set_display_name() sets the display-name property and emits a notify signal.

fn load_display_name(&self) {
  let priv_ = imp::Room::from_instance(&self);
  let matrix_room = priv_.matrix_room.get().unwrap().clone();
  do_async(
    async move { matrix_room.display_name().await },
    clone!(@weak self as obj => move |display_name| async move {
      match display_name {
        Ok(display_name) => obj.set_display_name(Some(display_name)),
        Err(error) => error!("Couldn't fetch display name: {}", error),
      };
   }),
  );
}

Important objects and widgets

Let’s have a look at some of the important GObjects and GtkWidgets we created. In some cases we create subclassed objects and in some cases we subclass widgets that keep track of the data produced by the SDK. This is because in many cases the data has only one consumer, the widget, so it doesn’t make sense to create a GObject and a GtkWidget that essentially contain the same data. Note that a widget is also a GObject. However, whenever we have information that is consumed by multiple widgets (e.g. a Matrix room) we create an object so that we don’t need to perform the same calls to the SDK multiple times and aggregate it for each single widget.

• Login is a widget that handles the login flow and produces a Session.
• The Session is the core of the client. It handles the login to the Matrix server and the sync, obviously via the matrix-rust-sdk. It also creates the UI and every widget needed to display a logged-in matrix account is a child of the Session or a submodule in the rust world.
• The Room is an object that stores all information pertaining to a Matrix room, e.g. the display name and timeline, and makes sure that the correct values are exposed as GObject properties.
  • The Timeline is a GListModel, the data structur behind GtkListView, containing all events that are shown in the room history.
• User is an object that tracks a Matrix user and exposes the display name and the avatar conveniently as GObject properties. To work properly it needs to receive room member state events. A User needs to be specific to each room because Matrix users can have different names and avatars for different rooms.
• Sidebar is the widget that contains the sidebar, build around a GtkListView.
• Content is the widgets to display a room’s content, including the room history. The room history uses a GtkListView that is connected to the Timeline.

All objects and widgets can be found in the Rust docs. Even though most entries don’t have any description yet, it should help you gain a deeper insight into the structure of the code.

Final words

Up to this point it was pretty much impossible to contribute to Fractal-Next because not much was set in stone and many parts still needed to be figured out. However, it’s now at the point where people can start contributing and help make Fractal-Next as awesome as the old Fractal, and of course way beyond that. We already have a few new contributors. To name two: Veli Tasalı, who worked on improving things around meson and added the generation of the rust docs to the CI, and Kévin Commaille who worked on improving the look and feel of the sidebar among other things.

Fractal-Next wouldn’t be possible without the awesome people working on the GTK Rust bindings, libadwaita, matrix-rust-sdk, and many more. Therefore a big thanks goes out to them. And of course, a huge thank you to NLnet and NEXT GENERATION INTERNET for sponsoring my work.

Fractal is a popular Matrix chat client for the GNOME desktop which has been in development since 2017. Fractal was designed to work well for collaboration in large groups, such as free software projects. However, there are still two main areas where it would benefit from improvements: its performance and maintainability are limited, and it lacks some important features such as end-to-end encryption (E2EE).

Adding E2EE will increase the confidence that users have in the privacy of their conversations, making it nearly impossible for their conversations to be accessed by others. Because E2EE aims to prevent the service provider from being able to decrypt the messages, because the encryption keys are stored only on the end-user’s device. The direct consequence of this is that some work is delegated to the client. Some of this functionality is the same for each and every Matrix client, and includes technical components that could easily be implemented in the wrong way (especially the various encryption and security features). Most security researchers agree that redoing this work is a bad idea as it can lead to vulnerabilities. More generally, reimplementing the same functionality for each client doesn’t make much sense. On the other hand, sharing it with others allows projects that use it to contribute their expertise and polish it together instead of competing on a multitude of implementations. That shared work is called an SDK and could be considered the future “engine” of Fractal.

When Fractal was created, there was no existing code that we could rely on. We had to implement ourselves bits of the Matrix protocol in Fractal, at a low level. In the meantime, the Matrix Foundation has kickstarted matrix-rust-sdk, a library to channel the Matrix community efforts into a common, efficient library. This library, still in development for now, will allow us to drop a lot of our own code.

The current Fractal engine, which handles all the interactions with the servers, is entangled with the parts that handle the user interface. This severely impairs the evolutionary potential of Fractal and has a significant impact on performance. At first we attempted to untangle the user interface and engine. Alejandro worked at the same time on refactoring our messy code and porting it to the SDK, but it was just too much of a hassle and breakage kept happening. We weighed the pros and cons and ended up deciding that a rewrite was the better alternative. We will also use this opportunity to update Fractal to use a newer version of GTK, the toolkit which Fractal uses for its user interface, which brings many performance improvements. The rewrite we decided upon is a perfect opportunity to transition to GTK4 and design for it.

We launched the Fractal-next initiative to carry out all of this work: rebuild Fractal from scratch, with improved foundations. Those foundations are GTK4 for the User Interface code, and matrix-rust-sdk for the engine code. Fractal-next is only just acquiring basic functionality so is not ready for general testing yet.

Sharing the work with others

The ultimate goal for the matrix-rust-sdk is to be feature-complete. It means the SDK aims at providing an easy way for developers to interact with servers using every feature offered by the Matrix protocol.

The SDK is still in its early stages, but is already feature-rich enough to be useful. In particular, it already implements the end-to-end encryption features we need to support in Fractal. We want to contribute to it by implementing the features we want to rely on for Fractal, but which are not exclusive to Fractal.

A major part that the SDK is lacking is a high-level API: a description of what the engine can do and how to use it. This would allow developers without knowledge of the underlying technology to work with Matrix. This API would translate simple calls such as “send this message to this person” into the actual technical action compliant with the Matrix protocol. In the Matrix world, almost everything is an event that is sent to a Room, a conceptual place to send and receive events from. Someone sends a message to a room: that’s an event. Someone changes their display name: that’s an event. Someone is kicked: you got it, that’s an event.

I already started working on designing and improving the API. So far I wrote the code that developers will use to interact with Matrix rooms and their membership. I also have plans to do some thing similar with Matrix events, but the design requires more discussion.

I also added some convenience methods to the API. The SDK can now easily retrieve the avatar for users and for matrix rooms. Future improvements include optimising the process for retrieving the avatar and caching the image so that it doesn’t need to be re-downloaded every time it is needed. This and other optimizations will happen behind the scenes inside the SDK, without requiring the application developer to do anything.

Code is only one aspect of contributions: a lot of the actual work consists of providing feedback and discussions around improving the code. I started the conversation about how the SDK should store Matrix events. So far, the SDK mostly stores and cares about end-to-end encryption events. The final goal will be that the SDK stores most of information locally, this will be especially crucial for the search in encrypted rooms since it’s performed locally because the server doesn’t have access to the content of a conversation.

Progress so far

Rewriting Fractal from scratch is a huge undertaking, but it has to happen and we are on the right path. While the contributions to matrix-rust-sdk are important as it is foundational work for Fractal, I could start working on the application itself too.

Most of the boilerplate is set up. That is quite generic code, which is mostly the same for every GTK app. This includes the the folder and module structure we will use in Fractal’s codebase. Now that the core structure is in place we can incrementally implement each piece of Fractal untill we reach a feature rich Matrix client. If you run Fractal-next right now you will already see most of the UI even though it’s pretty much an empty shell.

I already implemented the login via password in Fractal-next, but there is no single sign on support yet. This was done on purpose because the login flow of Matrix is expected to change in the near future and the current login is enough to start making use of matrix-rust-sdk.

Furthermore, I spent significant time on finding a good approach to connect Fractal and the matrix-rust-sdk. They are both written in Rust but Fractal relies heavily on GTK and GLib . As a result, they use quite different paradigms. The interconnection of Fractal and matrix-rust-sdk needs to be carefully thought through to make use of their full potentials.

After logging in to Fractal-next, you can currently see the list of your rooms. The sidebar is mostly complete, but doesn’t contain room categories such as Favorites yet. I will add the base for this feature to the SDK before I can implement it in Fractal.

I will cover the code of the sidebar and the rest of the architecture of Fractal-next in much more detail in a future blogpost.

Milestones and next steps

So what is next on my TODO list? Well, as I previously mentioned, we have a list of rooms, but no interaction with the items in this list are possible, yet. So the next step will be to add the room history and allow switching between rooms from within the sidebar. Clicking on a room should bring up the history for the selected room. Once this is implemented, we will add the possibility to send messages. And by this point Fractal-next will be huge step closer to a client, but will still miss a lot before it can be considered a feature rich client for instant messaging.

I would like to thank NLnet for funding my work on Fractal and Fractal-next, to build a feature rich and polished client to make the instant messaging experience on Matrix as pleasant as possible. The funding of NlNet allows me to focus also on the SDK, so that other people will be able to build great software on top of the SDK and us the Matrix ecosystem.

Some people already know, but now I’m officially announcing that for the next months I’ll be working full-time on Fractal thanks to a grant from NLnet. My main objective is to integrate end-to-end encryption into the GNOME Matrix client. Since user experience is crucial for getting E2EE right I’ll be working closely with Tobias Bernard from the design team throughout this project.

To give a rough roadmap, these are the main things I’m planning on working towards in the coming months:

1. Fully use and integrate matrix-rust-sdk
2. Device (Session) Management
   • List of active sessions
   • Logout (delete) from active sessions
   • View Session ID, Public Name, Last Seen
3. Conversation Encryption and Decryption: Allow sending and receiving messages. People need to be informed whether a conversation is encrypted or not, and have the possibility to enable encryption (disabling isn’t allowed by design).
4. User Verification: People can verify the identity of other Matrix users to set the trust level via emoji verification and QR code scanning.
5. Session Verification: People can verify their own sessions (cross singing) or choose not to trust cross signed sessions and manually sign other users’ individual sessions.
6. Key Backups (Secure Backup and Export Keys): Bundle all encryption keys and store them encrypted for backup.
7. Encrypted Room Search: Needs configuration options, and a local cache of encrypted messages for search.

I still need to figure out what the best approach regarding the planned GTK4 port (and other ongoing structural changes) is going to be, but I’ll be working with the rest of the Fractal team to find a solution, and hopefully get E2EE support into Fractal as soon as possible.

In the last couple of months, I was busy writing my thesis to conclude my master’s degree in computer science at the University of Bologna, therefore, I wasn’t much active in the GNOME community I hope that now I have much more time to dedicate to writing software ;).
The title of the thesis is ” Blockchain-based end-to-end encryption for Matrix instant messaging“. I researched an interesting experiment that uses an Ethereum based system to fully end-to-end encrypt a Matrix conversation.

Abstract

Privacy and security in online communication is an important topic today, especially in the context of instant messaging. A lot of progress has been made in recent years to ensure that conversations are secure against attacks by third parties, but privacy from the service provider itself remains difficult. There are a number of solutions offering end-to-end encryption, but most of them rely on a centralized server, proprietary clients, or both.
In order to have fully secure instant messaging conversations, a decentralized and end-to-end encrypted communication protocol is needed. This means there is no single point of control, and each message is encryped directly on the user’s device such that only the recipient can decrypt it.
This work proposes an end-to-end encryption system for the Matrix protocol based on blockchain technology. Matrix is a decentralized protocol and network for real-time communication that is currently mostly used for instant messaging. This protocol was selected because of its versatility and extensibility.
Using the Secret Store feature in OpenEthereum, the proposed system encrypts data using keys stored on the Ethereum blockchain. Access control to the keys is also handled by the Secret Store via a smart contract.
The proposed encryption system has multiple advantages over alternative schemes: The underlying blockchain technology reduces the risk of data loss because of its decentralized and distributed nature. Thanks to the use of smart contracts this system also allows for the creation of an advanced access control system to decryption keys.
In order to test and analyze the proposed design, a reference implementation was created in the form of a library. This library can be used for future research, but also as a building block for different applications to easily implement end-to-end encryption based on blockchain technology.

If you’re interested you can read the full thesis: Blockchain-based end-to-end encryption for Matrix instant messaging.

This announcement is long overdue, but better late than never 🙂

About 6 months ago I joined Purism, where I’m working on the Librem 5 phone. I’m in good company, since there are already a number of other fellow GNOME friends on the Librem 5 team, including Adrien Plazas, Tobias Bernard, and Mohammed Sadiq.

The last few months I’ve mainly been working on apps, but also a bit on phone shell (which is also written in GTK). I did a lot of work on Contacts (most of which is upstreamed in 3.36), the SMS and Calls apps, as well as Libhandy. Most recently I’ve been working on adding some initial quick settings toggles to the shell:

It’s been an awesome experience so far, because I get to work on a fully free software phone OS (which we really need, given the sorry state of Google-free Android) while also contributing to GNOME upstream.

For a University project a spent some time working on a project to digitally track the water consumption in my shared flat. Since nowadays everything is about data collection, I wanted to give this idea a shot. In my flat we have a simple analog water meter in my house.

Sadly, my meter is really dirt under the glass and i couldn’t manage to clean it. This will cause problems down the road.

The initial idea was easy, add a webcam on top of the meter and read the number on the upper half it. But I soon realized that the project won’t be that simple. The number shows only the use of 1m^3 (1000 liters), this means that I would have a change only every couple of days, which is useless and boring.  So, I had to read the analog gauges, which show the fraction in 0.0001, 0.001,  0.01 and 0.1 m^3. This discovery blocked me, and I was like “this is way to complicated”.

I have no idea how I found or what reminded me of OpenCV, but that was the solution. OpenCV is an awesome tool for computer vision, it has many features like Facial recognition, Gesture recognition … and also shape recognition. What’s a analog gauge? It’s just a circle with an triangular arrow indicating the value.

Let’s jump in to the project

I’m using a Raspberry Pi 1, a Logitech webcam, a juice bottle and some leds out of a bicycle light.

You need to find a juice bottle which fits nicely over the water meter. Cut of the top and bottom of the bottle and replace one side with a cardboard or wood with a hole in the middle. Attach the webcam centered over the hole and place a led on each side of the webcam to illuminate the water meter (you my need to cover them with paper to reduce reflection on the plastic of the meter).

First step is to set up the Raspberry Pi 1 (it doesn’t have to be a RPI, any computer running linux should work fine). You have to install a Linux Distro on the device, I used Archlinux. You can find a guide to install it on a Raspberry Pi 1 here.

After the initial setup you need to install git, python3 and opencv:
sudo pacman -S python3 git opencv
Clone the needed code to a know location:
git clone https://github.com/jsparber/water-meter-code.git
You need to create a new git repository to store the data and clone it to /home/alarm/water-meter-data/. If you want to use a different name or location you need to modify the name in measure.sh

On the RPI I have a cronjob which runs a script every minute. The script turns on the the led and then takes a picture then it turns the led off again, to save some energy.
With crontab -e you can modify the cron jobs, add * * * * * /home/alarm/code/take_photo.sh to run the take_photo.sh every minute, you may need to adjust the path depending on where you cloned the git repo.

After the picture is taken it calls a second script which then uses OpenCV to read the gauges and it appends the found values to a file which then is pushed to  git repo. I had a issue with the webcam. After some time my script couldn’t access the webcam anymore, I solved it by rebooting the RPI when it wasn’t possible to take a picture. (I did a quick search on the internet, most people solved this issue by changing the cam)

A nice optional feature is the home made switch connected to the RPI on the above picture. The schematics are really simple it’s only a 1KOhm resistor, a transistor and a USB extension cable. The transistor is switched on via the GPIO pin 18 of the Raspberry PI and gives power to the connected USB device. In this case I used it to connect the Leds.

Inside the USB extension cable there should be 4 different colored cables. We need to cut only the red one and connect it the same way as the schematics above show it, where the red_in goes to the male connector and the red_out to the female side of the cable. The GND needs to be connected to the ground pin of the Raspberry Pi, if you need to power something which requires more then 500mA you should connect the ground directly to the power source the same way as you did with the +5V red cable. You need to use the same power source for the switch and the RPI or it may not work.

And now the OpenCv part

First my code finds the circles of the right size on the image, and uses the two most left ones as gauges for 0.1 m^3 and 0.01 m^3 (Sadly since my meter is so dirty I can’t reliably read the other two values).

As the second step I create a mask which filters out everything what’s not red (remember the arrows are read). I take the contour of the mask which encloses the center of the circle I want to read. Then it finds the fairest point from the center of the circle which is the tip of the arrow. The software then creates a virtual line between the center and the tip, which is then used to calculate the angle which is basically the value shown on the gauge. The same thing is repeated for the other gauges.

This system sounds extremely simple, but to make everything work well together it isn’t that easy. OpenCV requires a lot of tuning, e.g selecting the right red color so that it detects it well but stays working even with light changes.

Conclusions

I learned a lot during this project, especially about OpenCV which i never used before. Sadly my water meter was really dirty so a couldn’t read all values and get also some wrong readings. So far I didn’t decide for what i want to use the collected data therefore I didn’t spend much time on finding a solution for read errors and problems I have when the gauges make a full turn. A easy solution would be to just keep an internal count of the water. And when we are unsure about a value we can go back to the memorized value.

The final plot can be found here. All values are saved directly without filter this causes the plot to have quite some noise but it allows to change the function used to filter later and adapt it to future needs.

My code is published on github:

• Scrips and source code
• Webpage with a plot of some data

Some sources which helped me a lot, many thanks to them:

• Python-cv-samples
• invisibility cloak using color detection and segmentation with opencv
• Analog-Clock-Reader
• Automating Picture Capture Using Webcams on Linux/Ubuntu
• Raspberry Pi – Driving a Relay using GPIO

This November I was in Barcelona for the Linux App Summit 2019. It was awesome \o/. I really liked that the conference was a joint event by GNOME and KDE, I met so many cool new people. During the conference I volunteered to show the “time left” signs to speakers, and helped out at the registration desk.

Aside from normal conference stuff I also managed to do quite a bit of hacking during the week. I made my first contribution to Gnome Initial Setup, and cleaned up Teleport a bit so I can hopefully get a new release out soon.

I’m bad at taking pictures, so here’s a picture of a tree in the middle of the stairs on the slopes of Mount Montjuic.

Thanks to the GNOME Foundation and Purism for sponsoring my travel and accommodation. The whole event was a lot of fun, so my thanks also go to all the organizers and people who helped make it happen!