Talking at mrmcds 2016 in Darmstadt, Germany

A couple of weeks ago, I attended the mrmcds in Darmstadt, Germany. Just like I did the last years. Like the years before, the conference was nicely themed. This year, the theme was all things medical. So speakers were given doctors’ coats, conference staff were running around like surgeons, alcohol could be had intravenously …

mrmcd 2016 logo

The talk on medical device nightmares (video) showed some medical devices like which show and record vital signs such as the pulse or blood pressure. But also more fancy devices such as an MRI. Of course, he did not only show the devices themselves, but rather how they tested them on their external interfaces, i.e. the networking port. Speaking of the MRI: It exposed a few hundred open ports. The services listening on these ports crashed when nmap scanned the host… But at least apparently they recovered automatically. He also presented an anaesthetic monitoring device, which is supposed to show how much alive a patient still is. The device seems to have a telnet interface which you can log on to with default credentials. The telnet interface has, not surprisingly, a command injection vulnerability, which allowed them to take ownership of the device. The next step was then to hijack the framebuffer and to render whatever they wanted on it. For example nice looking vital data; as if the patient was still alive. Or, probably the more obvious thing to do: Show Rick Astley.

It’s been an entertaining talk which makes you realise how complicated the whole area of pharmaceutical or medical appliances is. They need to go through a long and troublesome certification process, not unlike other businesses (say, car manufacturers). Patching the underlying Windows is simply not possible without losing the certification. You may well ask whether a certificate or an up-to-date OS is better for your health. And while I make it look a bit ridiculous now, I do appreciate that it’s a tough subject.

My own talk on GNOME (video) was well visited. I explained why I think GNOME is a good candidate for shipping security software to the masses. I said that GNOME cares about its users and goes the extra mile to support as many users as possible. That includes making certain decisions to provide a secure by default system. I gave two examples of how I think GNOME pushes the envelope when it comes to making security usable. One was the problem of OpenPGP Keysigning. I mentioned that it’s a very geeky thing which mortals do not understand. Neither do many security people, to be honest. And you can’t even blame them because it’s a messy thing to do. Doing it properly™ involves a metric ton of OpSec to protect the integrity of the key to be signed. I think that we can make the process much more usable than it is right now while even maintaining security. This year, I had Andrei working with me to make this happen.

The other example I gave was the problem of USB security. Do you know when you use your USB? And do you know when you don’t? And do you know when other people use your USB? I talked about the possibility to lock down your USB ports while you’re not in front of your computer. The argument goes that you can’t possibly insert anything if you’re away. Of course, there are certain cases to keep in mind, like not forbidding a keyboard to be plugged in, in case the old one breaks. But there is little reason to allow your USB camera to work unless you are actively using your machine. I presented how this could look like by showing off the work the George did last summer.

My friend Jens talked about Reverse Engineering of applications. He started to explain why you would do that in first place. Analysing your freshly received malware or weaknesses (think backdoors or bypasses) in your software are motivations, he said. But you might as well tinker with old software which has no developer anymore or try to find APIs of other software for interoperational purposes, he said. Let me note that with Free Software, you wouldn’t have to reverse engineer the binary ;-) But he also mentioned that industrial espionage is a reason for people to reverse engineer a compiled programme. The tool he uses the most is the “file” tool. He went on to explain the various executable formats for various machine flavours (think: x86, ELF, PE, JVM). To go practical, he showed a .NET application which only writes “hello, world!”, because malware, he said, is written in .NET nowadays. In order to decompile the binary he recommended “iLspy” as a one-stop suite for reverse engineering .NET applications. Next up were Android applications. He showed how to pull the APK off the device and how to decompose it to JAR classes. Then he recommended CFR for decompiling those into Java code. His clients, mostly banks, he said, try to hide secret keys in their apps, so the first thing he does when having a new job is to grep for “secret”. In 80% of the cases, he said, it is successful. To make it harder for someone to reverse engineer the binary, obfuscators exist for Java, but also for C. He also mentioned some anti debugging techniques such as to check for the presence of certain DLLs or to throw certain interrupts to determine whether the application runs under a debugger. It was a very practical talk which certainly made it clear that the presented things are relevant today. Due to the limited time and the many examples, he could only scratch the surface, though.

It’s been a nice conference with 400ish attendees. I really like how they care about the details, also when it comes to make the speakers feel good. It’s too sad that it’s only one weekend. I’m looking forward to attending next year’s edition :-)

(Re)mastering a custom Ubuntu auto-install ISO

Recently, I had to install GNU/Linux on a dozen or so machines. I didn’t want to install manually, mainly because I was too lazy, but also because the AC in the data centre is quite strong and I didn’t want to catch a cold… So I looked for some lightweight way of automatically installing an Ubuntu or so. Fortunately, I don’t seem to be the first person to be looking for a solution, although, retrospectively, I think the tooling is still poor.

I would describe my requirements as being relatively simple. I want to turn one of the to be provisioned machines on, wait, and then be able to log in via SSH. Ideally, most of the software that I want to run would already be installed. I’m fine with software the distribution ships. The installation must not require the Internet and should just work™, i.e. it should wipe the disk and not require anything special from the network which I have only little control over.

I looked at tools like Foreman, Cobbler, and Ubuntu’s MAAS. But I decided against them because it doesn’t necessarily feel lightweight. Actually, Cobbler doesn’t seem to work well when run on Ubuntu. It also fails (at least for me) when being behind an evil corporate proxy. Same for MAAS. Foreman seems to be more of a machine management framework rather than a hit and run style of tool.

So I went for an automated install using the official CD-ROMs. This is sub-optimal as I need to be physically present at the machines and I would have preferred a non-touch solution. Fortunately, the method can be upgrade to delivering the installation medium via TFTP/PXE. But most of the documents describing the process insist on Bind which I dislike. Also, producing an ISO is less error-prone so making that work first should be easier; so I thought.

Building an ISO

The first step is to mount to ISO and copy everything into a working directory. You could probably use something like isomaster, too.

mkdir iso.vanilla
sudo mount -oloop ubuntu.iso ./iso.vanilla
sudo cp -ar ./iso.vanilla/* ./iso.vanilla/.*

After you have made changes to your image, you probably want to generate a new ISO image that you can burn to CD later.

sudo mkisofs -J -l -b isolinux/isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table -z -iso-level 4 -c isolinux/ -o /tmp/ubuntu-16.04-myowninstall-amd64.iso -joliet-long

You’d expect that image to work If you now dd it onto a pendrive, but of course it does not… At least it didn’t for me. After trying many USB creators, I eventually found that you need to call isohybrid.

sudo isohybrid /tmp/ubuntu-16.04-myowninstall-amd64.iso

Now you can test whether it boots with qemu:

qemu-img create -f qcow2 /tmp/ubuntu.qcow2 10G
qemu-system-x86_64 -m 1G -cdrom ubuntu-16.04-server-amd64.iso -hda /tmp/ubuntu-nonet.qcow2

If you want to test whether a USB image would boot, try with -usb -usbdevice disk:/tmp/ubuntu-16.04-myowninstall-amd64.iso. If it doesn’t, then you might want to check whether you have assigned enough memory to the virtual machine. I needed to give -m 1G, because the default didn’t work with the following mysterious error.

Error when running with too little memory

It should also be possible to create a pendrive with FAT32 and to boot it on EFI machines. But my success was limited…

Making Changes

Now what changes do you want to make to the image to get an automated installation?
First of all you want to get rid of the language selection. Rumor has it that

echo en | tee isolinux/lang

is sufficient, but that did not work for me. Replacing timeout values in files in the isolinux to something strictly positive worked much better for me. So edit isolinux/isolinux.cfg.

If the image boots now, you don’t want the installer to ask you questions. Unfortunately, there doesn’t seem to be “fire and forget” mode which tries to install as aggressively as possible. But there are at least two mechanisms: kickstart and preseed. Ubuntu comes with a kickstart compatibility layer (kickseed).

Because I didn’t know whether I’ll stick with Ubuntu, I opted for kickstart which would, at least theoretically, allow me for using Fedora later. I installed system-config-kickstart which provides a GUI for creating a kickstart file. You can then place the file in, e.g. /preseed/ks-custom.cfg next to the other preseed files. To make the installer load that file, reference it in the kernel command line in isolinux/txt.cfg, e.g.

default install
label install
menu label ^Install Custom Ubuntu Server
kernel /install/vmlinuz
append file=/cdrom/preseed/ubuntu-server.seed vga=788 initrd=/install/initrd.gz ks=cdrom:/preseed/ks-custom.cfg DEBCONF_DEBUG=5 cdrom-detect/try-usb=false usb_storage.blacklist=yes --

Ignore the last three options for now and remember them later when we talk about issues installing from a pen drive.

When you boot now, you’d expect it to “just work”. But if you are me then you’ll run into the installer asking you questions. Let’s discuss these.

Multiple Network Interfaces

When you have multiple NICs, the installer apparently asks you for which interface to use. That is, of course, not desirable when wanting to install without interruption. The documentation suggest to use

d-i netcfg/choose_interface select auto

That, however, seemed to crash the installer when I configured QEMU to use four NICs… I guess it’s this bug which, at least on my end, had been cause by my accidentally putting “eth0” instead of “auto”. It’s weird, because it worked fine with the single NIC setup. The problem, it seems, is that eth0 does not exist! It’s 2016 and we have “predictable device names” now. Except that we still have /dev/sda for the first harddisk. I wonder whether there is a name for the first NIC. Anyway, if you do want to have the eth0 scheme back, it seems to be possible by setting biosdevname=0 as kernel parameter when booting.


You can test with multiple NICs and QEMU like this:

sudo qemu-system-x86_64 -m 1G -boot menu=on -hda /tmp/ubuntu-nonet.qcow2 -runas $USER -usb -usbdevice disk:/tmp/ubuntu-16.04-myowninstall-amd64.iso -netdev user,id=network0 -device e1000,netdev=network0 -netdev user,id=network1 -device e1000,netdev=network1 -netdev user,id=network2 -device e1000,netdev=network2 -netdev user,id=network3 -device e1000,netdev=network3 -cdrom /tmp/ubuntu-16.04-myowninstall-amd64.iso

No Internet Access

When testing this with the real servers, I realised that my qemu testbed was still too ideal. The real machines can resolve names, but cannot connect to the Internet. I couldn’t build that scenario with qemu, but the following gets close:

sudo qemu-system-x86_64 -m 1G -boot menu=on -hda /tmp/ubuntu-nonet.qcow2 -runas $USER -usb -usbdevice disk:/tmp/ubuntu-16.04-myowninstall-amd64.iso -netdev user,id=network0,restrict=y -device e1000,netdev=network0 -netdev user,id=network1,restrict=y -device e1000,netdev=network1 -netdev user,id=network2,restrict=y -device e1000,netdev=network2 -netdev user,id=network3,restrict=y -device e1000,netdev=network3 -cdrom /tmp/ubuntu-16.04-myowninstall-amd64.iso

That, however, fails:


The qemu options seem to make the built-in DHCP server to not hand out a default gateway via DHCP. The installer seems to expect that, though, and thus stalls and waits for user input. According to the documentation a netcfg/get_gateway value of "none" could be used to make it proceed. It’s not clear to me whether it’s a special none type, the string literal “none”, or the empty string. Another uncertainty is how to actually make it work from within the kickstart file, because using this debconf syntax is for preseeding, not kickstarting. I tried several things,

preseed netcfg/get_gateway none
preseed netcfg/get_gateway string
preseed netcfg/get_gateway string
preseed netcfg/get_gateway string none
preseed netcfg/no_default_route boolean true

The latter two seemed to worked better. You may wonder how I found that magic configuration variable. I searched for the string being displayed when it stalled and found an anonymous pastebin which carries all the configurable items.

After getting over the gateway, it complained about missing nameservers. By putting

preseed netcfg/get_nameservers string

I could make it proceed automatically.


Overwriting existing partitions

When playing around you eventually get to the point where you need to retry, because something just doesn’t work. Then you change your kickseed file and try again. On the same machine you’ve just left half-installed with existing partitions and all. For a weird reason the installer mounts the partition(s), but cannot unmount them


The documentation suggest that a line like

preseed partman/unmount_active boolean true

would be sufficient, but not so for me. And it seems to be an issue since 2014 at least. The workarounds in the bug do not work. Other sources suggested to use partman/early_command string umount -l /media || true, partman/filter_mounted boolean false, or partman/unmount_active seen true. Because it’s not entirely clear to me, who the “owner” , in terms of preseed, is. I’ve also experimented with setting, e.g. preseed --owner partman-base partman/unmount_active boolean true. It started to work when I set preseed partman/unmount_active DISKS /dev/sda and preseed --owner partman-base partman/unmount_active DISKS /dev/sda. I didn’t really believe my success and reordered the statements a bit to better understand what I was doing. I then removed the newly added statements and expected it to not work. However, it did. So I was confused. But I didn’t have the time nor the energy to follow what really was going on. I think part of the problem is also that it sometimes tries to mount the pendrive itself! Sometimes I’ve noticed how it actually installed the system onto the pendrive *sigh*. So I tried hard to make it not mount USB drives. The statements that seem to work for me are the above mentioned boot parameters (i.e. cdrom-detect/try-usb=false usb_storage.blacklist=yes) in combination with:

preseed partman/unmount_active boolean true
preseed --owner partman-base partman/unmount_active boolean true
preseed partman/unmount_active seen true
preseed --owner partman-base partman/unmount_active seen true

#preseed partman/unmount_active DISKS /dev/sda
#preseed --owner partman-base partman/unmount_active DISKS /dev/sda

preseed partman/early_command string "umount -l /media || true"
preseed --owner partman-base partman/early_command string "umount -l /media ||$

How I found that, you may ask? Enter the joy of debugging.

Debugging debconf

When booting with DEBCONF_DEBUG=5, you can see a lot of information in /var/log/syslog. You can see what items are queried and what it thinks the answer is. It looks somewhat like this:


You can query yourself with the debconf-get tool, e.g.

# debconf-get partman/unmount_active

The file /var/lib/cdebconf/questions.dat seems to hold all the possible items. In the templates.dat you can see the types and the defaults. That, however, did not really enlighten me, but only wasted my time. Without knowing much about debconf, I’ve noticed that you seem to be able to not only store true and false, but also flags like “seen”. By looking at the screenshot above I’ve noticed that it forcefully sets partman/unmount_active seen false. According to the documentation mentioned above, some code really wants this flag to be reset. So that way was not going to be successful. I noticed that the installer somehow sets the DISKS attribute to the partman/unmount_active, so I tried to put the disk in question (/dev/sda) and it seemed to work.

Shipping More Software

I eventually wanted to install some packages along with the system, but not through the Internet. I thought that putting some more .debs in the ISO would be as easy as copying the file into a directory. But it’s not just that easy. You also need to create the index structure Debian requires. The following worked well enough for me:

cd pool/extras
apt-get download squid-deb-proxy-client
cd ../..
sudo apt-ftparchive packages ./pool/extras/ | sudo tee dists/stable/extras/binary-i386/Packages

I was surprised by the i386 suffix. Although I can get over the additional apt-ftparchive, I wish it wouldn’t be necessary. Another source of annoyance is the dependencies. I couldn’t find a way to conveniently download all the dependencies of a given package.

These packages can then be installed with the %packages directive:

@ ubuntu-server

Or via a post-install script:


apt-get install -y squid-deb-proxy-client
apt-get update
apt-get install -y htop
apt-get install -y glusterfs-client glusterfs-server
apt-get install -y screen
apt-get install -y qemu-kvm libvirt-bin

Unfortunately, I can’t run squid-deb-proxy-client in the installer itself. Not only because I don’t know how to properly install the udeb, but also because it requires the dbus daemon to be run inside the to-be-installed system which proves to be difficult. I tried the following without success:

preseed anna/choose_modules string squid-deb-proxy-client-udeb

preseed preseed/early_command string apt-install /cdrom/pool/extras/squid-deb-proxy-client_0.8.14_all.deb

anna-install /cdrom/pool/extras/squid-deb-proxy-client-udeb_0.8.14_all.udeb

If you happen to know how to make it work, I’d be glad to know about it.

Final Thoughts

Having my machines installed automatically cost me much more time than installing them manually. I expected to have tangible results much quicker than I actually did. However, now I can re-install any machine within a few minutes which may eventually amortise the investment.

I’m still surprised by the fact that there is no “install it, dammit!” option for people who don’t really care about the details and just want to get something up and running.

Unfortunately, it seems to be non-trivial to just save the diff of the vanilla and the new ISO :-( The next Ubuntu release will then require me to redo the modifications. Next time, however, I will probably not use the kickseed compatibility layer and stick to the pure method.

Pressemitteilung: GUADEC startet ab morgen in Karlsruhe

(cross posting from which you should follow!)

Am 21.09.2016 wird das nächste GNOME Release (3.22) unter dem Namen “Karlsruhe” erscheinen. Genannt wird ein GNOME Release nach dem letzten Austrageort der GUADEC (GNOME Users And Developers European Conference). Während dieser Konferenz treffen sich jährlich zirka 200 Freie-Software-Enthusiasten zu Workshops, Vorträgen und Arbeitsgruppentreffen und arbeiten an GNOME, der Software-Lösung für alle. Dieses Jahr findet das Treffen vom 11.08. bis zum 17.08. am KIT in Karlsruhe statt.

Die Veranstaltung richtet sich sowohl an Entwickler als auch an Nutzer, und soll dazu dienen, Wissen zu erzeugen und weiter zu leiten. Wir freuen uns, die diesjährige GUADEC in Karlsruhe zu veranstalten. Karlsruhe steht für uns für einen starken wissenschaftlichen und technischen Hintergrund verbunden mit dem gewissen Etwas an Kreativität und Design. Diese Verbindung zwischen Technik und Design passt sehr gut zu dem GNOME Projekt, welches nicht nur auf technischer Seite seit Jahren neue Standards fördert, sondern auch im Bereich des User-Experience-Design innovative Wege geht.

GNOME, die Stiftung mit dem Fuß als Logo, steht für 3 Kernpunkte:


GNOME vereint nicht nur freiwillige und bezahlte Programmierer, sondern auch Firmen und wohltätige Organisationen. Wir machen GNOME 3, ein Computersystem das einfach zu benutzen ist, in über 190 Sprachen übersetzt wurde und großen Wert auf Behindertengerechtigkeit legt. Wir entwickeln in der Öffentlichkeit und jeder kann bei GNOME mitmachen. Unsere Kommunikationskanäle stehen allen offen, der Quellcode kann frei heruntergeladen, modifiziert und weitergegeben werden.


GNOME ist eine flexible und mächtige Plattform für Desktop- und mobile Anwendungen. Wir entwickeln Toolkits wie GTK+ oder Clutter, die Webbrowser Engine WebKitGTK+, TMMultimedia-Bibliotheken wie GStreamer, das D-Bus messaging system, oder die Pango Textrendering-Bibliothek. GNOME Entwickler gehen auch weiter und modifizieren essentielle Kern-Infrastruktur wie den Linux Kernel, systemd oder Wayland.


GNOME 3 kann über viele bekannte GNU/Linux-Distributionen wie Debian, Fedora, OpenSuSE oder Ubuntu bezogen werden. GNOME Technologien sind der Treiber vieler Firmen wie Endless Mobile, Amazon, TiVo, Nokia, TouchTune, Garmin oder TomTom. Viele andere Firmen benutzen GNOME Komponenten kostenfrei im Rahmen der Freien Lizenz unserer Software.

Wir laden gerne zu einem Treffen oder Interview während unserer Konferenz ein. Bei Fragen steht Tobias Mueller ( oder +4915153778790) gerne zur Verfügung.

Mehr Informationen über die GUADEC und GNOME gibt es unter:

GNOME Keysign 0.6

It’s been a while since I reported on GNOME Keysign. The last few releases have been exciting, because they introduced nice features which I have been waiting long for getting around to implement them.

So GNOME Keysign is an application to help you in the OpenPGP Keysigning process. That process will eventually require you to get hold of an authentic copy of the OpenPGP Key. In GNOME Keysign this is done by establishing a TCP connection between two machines and by exchanging the data via that channel. You may very well ask how we ensure that the key is authentic. The answer for now has been that we transmit the OpenPGP fingerprint via a secure channel and that we use the fingerprint to authenticate the key in question. This achieves at least the same security as when doing conventional key signing, because you get hold of the key either via a keyserver or a third party who organised the “key signing party”. Although, admittedly, in very rare cases you transfer data directly via a USB pendrive or so. Of course, this creates a whole new massive attack surface. I’m curious to see technologies like wormhole deployed for this use case.

The security of you going to the Internet to download the key is questionable, because not only do you leak that you’re intending to communicate with a certain person, but also expose yourself to attacks like someone dropping revocation certificates or UIDs of the key of your interest. While the former issue is being tackled by not going to the Internet in first place, the latter had not been dealt with. But these days are over now.

As of 0.5 GNOME Keysign also generates an HMAC of the data to be transferred and encodes that in the QR Code. The receiving end can then verify whether the data downloaded matches the expected value. I am confident that a new generation hash function serves the same purpose, but I’m not entirely sure how easy it is to get Keccak or siphash into the users’ hands. The HMAC, while being cryptographic overkill, should be fine, though. But the construction leaves a bad taste, especially because a known key is currently used to generate the HMAC. But it’s a mechanism built-in into Python. However, I expect to replace that with something more sensible.

In security, we better imagine a strong attacker who is capable of executing attacks which we think are not necessarily easy or even possible to mount. If we can defend against such a strong attacker then we may trust the system to resist weaker attacks, too. One of such a difficult attack, I think, is to inject just one frame while, at the same time, controlling the network. The attack could then make the victim scan a rogue barcode which delivers a rogue MAC which in turn validates the wrong data. Such an attack should not go unnoticed and, as of 0.5, GNOME Keysign will display the frame that contained the barcode.

This is what it looked like before:


And now you can see the frame that got decoded. This is mainly because the GStreamer zbar element also provides the frame.


Another interesting feature is the availability of a separate tool for producing signatures for a given key in a file. The scenario is that you may have received a key from your friend via a (trusted, haha) pendrive, a secure network connection (like wormhole), or any other means you consider sufficiently integrity preserving. In order to sign that key you can now execute something like python -m keysign.gnome-keysign-sign-key in order to run all the signing logic but without the whole key transfer stuff. This is a bit experimental though and I am not yet happy about the state that program is in, so it’s not directly exposed to users by installing it as executable.

GNOME Keysign is available in OpenSuSE, now. I don’t know the exact details of how to make it work, but rumour has it that you can just do a zypper install gnome-keysign. While getting there we identified a few issues along the way. For example, the gstreamer zbar element needs to be present. But that was a problem, because the zbar element was not built because the zbar library was not available. So that needed to get in first. Then we realised that the most modern OpenSuSE uses a very recent GnuPG which the currently used GnuPG library is not handling so nicely. That caused a few headaches. Also, the firewall seems to be an issue which needs to be dealt with. So much to code, so little time! ;-)

Talking at OpenSuSE Conference 2016 in Nuremberg

I was invited to this year’s OpenSuSE Conference in Nuremberg, Germany. I had been to that event two years ago in Dubrovnik which I enjoyed so much that I was eager to go again.


The venue was very easy to find due to poster hanging everywhere. The flow of information was good in general. That includes emails being every day which highlighted items in the schedule or restaurant recommendations for the evening.

I arrived just in time for my first show on GNOME Keysign. For better or worse we only very few people so we could discuss matters deeply. It was good, because we found bugs and other user facing issues that need to be resolved. The first and most obvious one was GnuPG 2.1 support. Although still experimental, OpenSuSE ships 2.1 by default. The wrapping library we’re using to interact with GnuPG did not support calling the newer gpg, so we had to identify the issues, find a fix, and test. It eventually worked out :-)

I also had a talk called “Five years after 3.0” which, to my surprise, has been covered by reddit and omgubuntu. I was also surprised by the schedule which only gave me 30 minutes instead of the usual 45 or 60. I was eventually politely reminded that I have significantly exceeded my time *blush*. We thus needed to move discussions outside which was fruitful. People at OpenSuSE Con are friendly and open-minded. It’s a pleasure to have arguments there :)

I didn’t actually see many talks myself. Although the schedule was quite full with interesting topics! But knowing that the VoCCC people were running the video recordings, I could count on recordings being available after a few days hours.

But I have had very interesting and enlightening discussions about distributions, containerised apps, Open Build Service, OpenQA, dragging more GNOME people towards OpenSuSE, Fonts, and other issues. That’s the great thing about conferences: You get to know people with interesting stories. As for the fonts, for example, I was discussing the complexity involved in rendering glyphs and whether this could eventually lead to security problems. I think the attack surface of fonts has been undervalued and needs some investigation. I hope I can invest some time in looking at building and modifying fonts. I also found it interesting to discuss why I would not recommend OpenSuSE as a GNU/Linux distribution to anyone, mainly because I need to reflect and challenge myself. Turns out, I don’t have any good reason except that my habits simply don’t include using OpenSuSE myself and I am thus unable to give a recommendation. I think they have interesting infrastructure though. I see the build service for having peoples’ apps built and OpenQA for having them tested. Both seem to be a little crude overall, but could become the tools to use for distributing your flatsnappimgpack. An idea was circling around to have a for those app image formats and execution environments. But in a somewhat more working state. I think key to success of any such body is being lightweight and not end up like openstack. Let’s hope we can bring people who work on various parts or even implementation of containerisation for desktop applications together. I also hope that the focus for containered desktop apps will be isolation from other apps rather than actually distributing the software, because I don’t think we have a big problem with getting Free Software into the user’s hands.

So a big “thank you” to this year’s organisers for this event. I hope I can attend on of the following conferences :)

Talking at GNOME.Asia Summit 2016 in New Delhi, India

It’s spring time and that means it’s time for GNOME.Asia Summit! This year’s edition took place in New Delhi, India. This years makes five years after the initial GNOME 3.0 release. In fact, an important releases planning hackfest happened five years ago in India, so it’s been a somewhat remarkable date.

The conference felt a little smaller than the last edition, although I guess the college we were hosted at tried hard to bring their students to the talks. That was especially noticeable in the opening slot were everybody who felt sufficiently important had something to say. The big auditorium was filled with students, but I doubt they were really interested or listening. The opening was a bit weird for my taste, anyway. I have seen many conference openings, I would say. But that guy from the college who opened GNOME.Asia 2016 seemed to be a little bit confused, I have the feeling. He said that GNOME started 2008 so that all the software you use can be had freely so that you can upgrade your devices, like GPS satnavs. The opening ceremony, and yes, it’s really more of a ceremony rather than a short “welcome, good that you’re here” talk seems to be quite a formal thing in this college. Everybody on the stage receives a bouquet of flowers and many people were greeted and saluted to which stretched everything to an enormous length which in turn made the schedule slip by two hours or so.

Cosimo keynoted the conference and presented his ideas for the future of the GNOME project. We’ve come a long way, he said, with GNOME 3, which has initially been released five years ago. GNOME has aged well, he said. No wrinkles can be seen and GNOME is looking better than ever. He said that he likes GNOME 2 to be thought of GNOME in the era of distributions, because you could plug together modules that you liked. And everybody liked that. The pain point, he said, was that distributions chose which modules to plug together which finally decided about the user experience. Due to module proliferation was felt as impacting the project negatively the new world of GNOME 3 was introduced. One the most controversial but also most successful thing GNOME 3 did, he said, was to put the responsibility of defining the user experience back in upstream’s hands by eliminating choices. While causing people to complain, it led to a less complicated test matrix which eventually made GNOME accessible to less technical people. He said, GNOME 3 is the era of Operating Systems, so there are not distributions packaging GNOME but rather Operating systems built on GNOME, like endless, mint, or solus. The big elephant in the room is the role of applications, he said. If cohesive Operating Systems are built upon GNOME, how can applications work with different operating systems? Currently, you cannot, he said, run elementary applications on GNOME and vice versa. xdg-app will hopefully address that, he said. It’s a big transition for the GNOME project and that transition is even bigger than the one from GNOME 2 to GNOME 3, he said. Unfortunately, the audience seemed to be a little tired by from the length of the opening session and it felt like they were demanding a break by starting to chat with their neighbours…

Pravin then continued to talk about the state of Indian languages in GNOME. He mentioned that some Indian languages are well supported while some others have no support at all. He also showed that with Fedora 24 you get a text prediction engine. So you can type Latin characters for the word you want to enter in a different script. The Q&A revealed that the list of suggested words is sorted by frequency. Apparently they did some analysis of usage of words. I wonder whether it’s also able to learn from the user’s behaviour.

The talk on privacy given by Ankit Prateek showed how your typical Internet and Web usage would leave traces and what you mitigation you could employ. He mentioned specific Web attacks like Super Cookies or Canvas fingerprinting. He recommended using NoScript whichs usefulness the audience immediately questioned. To my surprise, he didn’t mention one my favourite plugins Google Privacy, because Google remembers what search results you click.

I got to talk about five years of GNOME 3. I conveyed the story of how the 3.0 release happened and what was part of it. For example, we had so many release parties with swag being sent around the world! But I also showed a few things that have changed since the initial 3.0.

Another talk I had was about Security. I explained why I see GNOME being in the perfect position to design, develop, and deploy security systems for a wide range of users. First, I ranted about modal dialogues, prompts, and that they are not a good choice for making a security decision. Then, I explained how we could possible defend against malicious USB devices. I think it’s work we, as developers of a Free Software desktop, have to do in order to serve our users. Technically, it’s not very hard, e.g. you block new USB devices being plugged in, when the screensaver is shown. We know how to do the blocking and unblocking of USB devices. More subtle issues involve the policies to apply and how to make the user aware of USB devices. Another pet peeve of mine is Keysigning, so I also ranted about the state of the art and we can and should improve things.

Thanks to the local organising people and the GNOME Foundation for flying me in and out.
Sponsored by GNOME!

Talking at FOSSASIA 2016 in Singapore

This year I was able to attend this year’s FOSSASIA in Singapore. It’s quite a decently sized event with more than 150 speakers and more than 1000 people attending. Given the number of speakers you can infer that there was an insane number of talks in the two and a half day of the conference. I’ve seen recordings being made so I would expect those to show up at some stage, but I don’t have any details. The atmosphere was very friendly and the venue a-maze-ing. By that I mean that it was a fantastic and huge maze. We were hosted in Singapore’s Science Museum which exhibits various things around biology, physics, chemistry, and much more. It is a rather large building in which it was easy to get lost. But it was great being among those sciency exhibits and to exchange ideas and thoughts. Sometimes, we could see an experiment being made as a show to the kids visiting the museum. These shows included a Tesla coil or a fire tornado. Quite impressive.

One of the first things I could see was Cat Altmann talking about her position being in the field of marketing which, she says, engineers tend to not like. But as opposed to making people like things they don’t need or want, she is rather concerned with reaching out to people to open source code. The Making and Science team within Google exists to work on things like sending kids from under resourced schools to field trips. Science also plays a role in this year’s Summer of Code. 43+ out of 180 projects are related to science, she said.

Nikolai talked about the Nefertiti hack. In case you missed it, the Nefertiti was “cloned”. The bust is a 3000 years old artefact which is housed in Berlin and is publicly available (for people who can travel to Berlin…). The high resolution data of the scanned object is, however, not available, along with many many other data that the museums have about their objects. He compared that behaviour to colonisation; I couldn’t really follow why, though. Anyway, they managed to scan the bust themselves by sneaking into the museum and now they’ve released the data. Their aim, as far as I could follow, was to empower people to decide about what culture is and what not. Currently, it is the administration which decides, he said. With the data (and I think with a printed copy of the bust) they travelled to Egypt to make an exhibition. But beforehand, they had produced a video which substantiates the claim of having found a second bust while digging for artefacts. The talk itself was interesting, but the presentation was bad. the speaker was lost a few times and didn’t know how to handle the technical side of the presentation. Anyway, I like it when such guerrilla art makes it into the news.

Lenny talked about systemd which uncovered some news that you may have missed if you’re not following its development too closely. He said that systemd has moved to github and while it’s attracting new contributors it still has major issues which he didn’t mention though. A component named networkd is now the default on both Fedora and Ubuntu. It’s a rather underwhelming piece of software though, because it has no runtime interface. The nspawn tool is used by CoreOS’ rkt docker alternative. He also mentioned sd-bus which he claims is a replacement for the reference DBus implementation. Another interesting thing he mentioned is that systemd can not only do socket activation but also USB Function FS activation. So whenever you are in the need to start your USB gadget only when the USB cable has been plugged in, systemd may be for you.

In another session, Lenny continued talking about systemd with regards to its container capabilities. Containers are all the rage, right..? He said that all the systemd tools work on containers, too, with the -M switch. systemd also just works inside a container, with the exception of Docker, he said. It is also possible to make systemd download and verify images to run full system images. Funnily enough, he said, Ubuntu images are properly signed but Fedora images are not.

I also had a talk and a workshop to give. The workshop was titled “Functionality, Security, Usability: Choose any two. Or GNOME.” which is a bit sensational, I admit. I’m not experienced with holding workshops and it was unclear to me what to expect. Workshop sounds to me like people come and they want to hack on something. The venue, however, was not necessarily equipped with a reliable Internet connection for the attendees. Also, the time was set to one hour. I don’t think you can do a meaningful workshop within one hour. so I didn’t really know how to prepare. I ended up ranting about OpenPGP, GnuPG, and SKS. Then I invited people to hack on GNOME Keysign which was a bit difficult. given the time constraints we had. Well, that I mainly had, because I was meant to give a proper talk shortly after.

During my talk, I gave a glimpse on what to expect from GNOME 3.20 codename Delhi. It was a day before the release, so it was the perfect timing for getting people excited. And I think it worked reasonably well. I would have loved to be able to show the release video, but it wasn’t finished until then. So I mainly showed screenshots of the changes and discussed on a high level what GNOME is and what it is not. People were quite engaged and still believe GNOME 3 was designed for tablets.

FOSSASIA used to be in Vietnam and it was actually co-hosted with GNOME.Asia Summit once. It smells like we could see such a double event in the future, but probably in Singapore. I think that’d be great, because FOSSASIA is a well organised event, albeit a little chaos here and there. But who doesn’t have that… In fact, I nearly couldn’t make it to the conference, because GNOME did not react for two weeks so the conference removed me from the schedule. Eventually, things worked out, so all is good and I would like to thank the GNOME Foundation for contributing to the coverage of the costs.

Sponsored by GNOME!

Taint Tracking for Chromium

I forgot to blog about one of my projects. I had actually already talked about it more than one year ago and we had a paper at USENIX Security.

Essentially, we built a protection against DOM-based Cross-site Scripting (DOMXSS) into Chromium. We did that by detecting whenever potentially attacker provided strings become JavaScript code. To that end, we made the HTML rendering engine (WebKit/Blink) and the JavaScript engine taint aware. That is, we identified sources of values that an attacker could control (think and marked all strings coming from those sources as tainted. Then, during parsing of JavaScript, we check whether the string to be compiled is actually tainted. If that is indeed the case, then we abort the compilation.

That description is a bit simplified. For example, not compiling code because it contains some fragments of the URL would break a substantial number of Web sites. It’s an unfortunate fact that many Web sites either eval code containing parts of the URL or do a document.write with a string containing parts of the URL. The URL, in our attacker model, can be controlled by the attacker. So we must be more clever about aborting compilation. The idea was to only allow literals in JavaScript (like true, false, numbers, or strings) to be compiled, but not “code”. So if a tainted (sub)string compiles to a string: fine. If, however, we compile a tainted string to a function call or an operation, then we abort. Let me give an example of an allowed compilation and a disallowed one.



var pos=document.URL.indexOf("name=")+5;

Welcome to our system


Which is from the original report on DOM-based XSS. You see that nothing bad will happen when you open However, opening will lead to attacker provided code being executed in the victim’s context. Even worse, when opening with a hash (#) instead of a question mark (?) then the server will not even see the payload, because Web browsers do not transmit it as part of their request.

“Why does that happen?”, you may ask. We see that the document.write call got fed a string derived from the URL. The URL is assumed to be provided by the attacker. The string is then used to create new DOM elements. In the good case, it’s only a simple text node, representing text to be rendered. That’s a perfectly legit use case and we must, unfortunately, allow that sort of usage. I say unfortunate, because using these APIs is inherently insecure. The alternative is to use createElement and friends to properly inject DOM nodes. But that requires comparatively much more effort than using the document.write. Coming back to the security problem: In the bad case, a script element is created with attacker provided contents. That is very bad, because now the attacker controls your browser. So we must prevent the attacker provided code from execution.

You see, tracking the taint information is a non-trivial effort and must be done beyond newly created DOM nodes and multiple passes of JavaScript (think eval(eval(eval(tainted_string)))). We must also track the taint information not on the full string, but on each character in order to not break existing Web applications. For example, if you first concatenate with a tainted string and then remove all tainted characters, the string should not be marked as tainted. This non-trivial effort manifests itself in the over 15000 Lines of Code we patched Chromium with to provide protection against DOM-based XSS. These patches, as indicated, create, track, propagate, and evaluate taint information. Also, the compilation of JavaScript has been modified to adhere to the policy that tainted strings must only compile to literals. Other policies are certainly possible and might actually increase protection (or increase compatibility without sacrificing security). So not only WebKit (Blink) needed to be patched, but also V8, the JavaScript engine. These patches add to the logic and must be execute in order to protect the user. Thus, they take time on the CPU and add to the memory consumption. Especially the way the taint information is stored could blow up the memory required to store a string by 100%. We found, however, that the overhead incurred was not as big as other solutions proposed by academia. Actually, we measure that we are still faster than, say, Firefox or Opera. We measured the execution speed of various browsers under various benchmarks. We concluded that our patched version added 23% runtime overhead compared to the unpatched version.


As for compatibility, we crawled the Alexa Top 10000 and observed how often our protection mechanism has stopped execution. Every blocked script would count towards the incompatibility, because we assume that our browser was not under attack when crawling. That methodology is certainly not perfect, because only shallowly crawling front pages does not actually indicate how broken the actual Web app is. To compensate, we used the WebKit rendering tests, hoping that they cover most of the important functionality. Our results indicate that scripts from 26 of the 10000 domains were blocked. Out of those, 18 were actually vulnerable against DOM-based XSS, so blocking their execution happened because a code fragment like the following is actually indistinguishable from a real attack. Unfortunately, those scripts are quite common :( It’s being used mostly by ad distribution networks and is really dangerous. So using an AdBlocker is certainly an increase in security.

var location_parts = window.location.hash.substring(1).split(’|’);
var rand = location_parts[0];
var scriptsrc = decodeURIComponent(location_parts[1]);
document.write("<scr"+"ipt src=’" + scriptsrc + "’></scr"+"ipt>");

Modifying the WebKit for the Web parts and V8 for the JavaScript parts to be taint aware was certainly a challenge. I have neither seriously programmed C++ before nor looked much into compilers. So modifying Chromium, the big beast, was not an easy task for me. Besides those handicaps, there were technical challenges, too, which I didn’t think of when I started to work on a solution. For example, hash tables (or hash sets) with tainted strings as keys behave differently from untainted strings. At least they should. Except when they should not! They should not behave differently when it’s about querying for DOM elements. If you create a DOM element from a tainted string, you should be able to find it back with an untainted string. But when it comes to looking up a string in a cache, we certainly want to have the taint information preserved. I hence needed to inspect each and every hash table for their usage of tainted or untainted strings. I haven’t found them all as WebKit’s (extensive) Layout tests still showed some minor rendering differences. But it seems to work well enough.

As for the protection capabilities of our approach, we measured 100% protection against DOM-based XSS. That sounds impressive, right? Our measurements were two-fold. We used the already mentioned Layout Tests to include some more DOM-XSS test cases as well as real-life vulnerabilities. To find those, we used the reports the patched Chromium generated when crawling the Web as mentioned above to scan for compatibility problems, to automatically craft exploits. We then verified that the exploits do indeed work. With 757 of the top 10000 domains the number of exploitable domains was quite high. But that might not add more protection as the already existing built in mechanism, the XSS Auditor, might protect against those attacks already. So we ran the stock browser against the exploits and checked how many of those were successful. The XSS Auditor protected about 28% of the exploitable domains. Our taint tracking based solution, as already mentioned, protected against 100%. That number is not very surprising, because we used the very same codebase to find vulnerabilities. But we couldn’t do any better, because there is no source of DOM-based XSS vulnerabilities…

You could, however, trick the mechanism by using indirect flows. An example of such an indirect data flow is the following piece of code:

// Explicit flow: Taint propagates
var value1 = tainted_value === "shibboleth" ? tainted_value : "";
// Implicit flow: Taint does not propagate
var value2 = tainted_value === "shibboleth" ? "shibboleth" : "";

If you had such code, then we cannot protect against exploitation. At least not easily.

For future work in the Web context, the approach presented here can be made compatible with server-side taint tracking to persist taint information beyond the lifetime of a Web page. A server-side Web application could transmit taint information for the strings it sends so that the client could mark those strings as tainted. Following that idea it should be possible to defeat other types of XSS. Other areas of work are the representation of information about the data flows in order to help developers to secure their applications. We already receive a report in the form of structured information about the blocked code generation. If that information was enriched and presented in an appealing way, application developers could use that to understand why their application is vulnerable and when it is secure. In a similar vein, witness inputs need to be generated for a malicious data flow in order to assert that code is vulnerable. If these witness inputs were generated live while browsing a Web site, a developer could more easily assess the severity and address the issues arising from DOM-based XSS.


It the beginning of the year and, surprise, FOSDEM happened :-) This year I even managed to get to see some talks and to meet people! Still not as many as I would have liked, but I’m getting there…

Lenny talked about systemd and what is going to be added in the near future. Among many things, he made DNSSEC stand out. I not sure yet whether I like it or not. One the one hand, you might get more confidence in your DNS results. Although, as he said, the benefits are small as authentication of your bank happens on a different layer.

Giovanni talked about the importance of FOSS in the surveillance era. He began by mentioning that France declared the state of emergency after the Paris attacks. That, however, is not in line with democratic thinking, he said. It’s a tool from a few dozens of years ago, he said. With that emergency state, the government tries to weaken encryption and to ban any technology that may be used by so called terrorists. That may very well include black Seat cars like the ones used by the Paris attackers. But you cannot ban simple tools like that, he said. He said that we should make our tools much more accessible by using standard FLOSS licenses. He alluded to OpenSSL’s weird license being the culprit that caused Heartbleed not to have been found earlier. He also urged the audience to develop simpler and better tools. He complained about GnuPG being too cumbersome to use. I think the talk was a mixed bag of topics and got lost over the many topics at hand. Anyway, he concluded with an interesting interpretation of Franklin’s quote: If you sacrifice software freedom for security you deserve neither. I fully agree.

In a terrible Frenglish, Ludovic presented on Python’s async and await keywords. He said you must not confuse asynchronous and parallel execution. With asynchronous execution, all tasks are started but only one task finishes at a time. With parallel execution, however, tasks can also finish at the same time. I don’t know yet whether that description convinces me. Anyway, you should use async, he said, when dealing with sending or receiving data over a (mobile) network. Compared to (p)threads, you work cooperatively on the scheduling as opposed to preemptive scheduling (compare time.sleep vs. asyncio.sleep).

Aleksander was talking on the Tizen security model. I knew that they were using SMACK, but they also use a classic DAC system by simply separating users. Cynara is the new kid on the block. It is a userspace privilege checker. A service, like GPS, if accessed via some form of RPC, sends the credentials it received from the client to Cynara which then makes a decision as to whether access is allowed or not. So it seems to be an “inside out” broker. Instead of having something like a reference monitor which dispatches requests to a server only if you are allowed to, the server needs to check itself. He went on talking about how applications integrate with Cynara, like where to store files and how to label them. The credentials which are passed around are a SMACK label to identify the application. The user id which runs the application and privilege which represents the requested privilege. I suppose that the Cynara system only makes sense once you can safely identify an application which, I think, you can only do properly when you are using something like SMACK to assign label during installation.

Daniel was then talking about his USBGuard project. It’s basically a firewall for USB devices. I found that particularly interesting, because I have a history with USB security and I do know that random USB devices pose a problem. We are also working on integrating USB blocking capabilities with GNOME, so I was keen on meeting Daniel. He presented his program, what it does, and how to use it. I think it’s a good initiative and we should certainly continue exploring the realm of blocking USB devices. It’s unfortunate, though, that he has made some weird technological choices like using C++ or a weird IPC system. If it was using D-Bus then we could make use of it easily :-/ The talk was actually followed by Krzyzstof who I reported on last time, who built USB devices in software. As I always wanted to do that, I approached him and complained about my problems doing so ;-)

Chris from wolfSSL explained how they do testing for their TLS implementation. wolfSSL is 10 years old and secures over 1 billion endpoints, he said. Most interestingly, they have interoperability testing with other TLS implementations. He said they want to be the most well tested TLS library available which I think is a very good goal! He was a very good speaker and I really enjoyed learning about their different testing strategies.

I didn’t really follow what Pam was talking about implicit trademark and patent licenses. But it seems to be an open question whether patents and trademarks are treated similarly when it comes to granting someone the right to use “the software”. But I didn’t really understand why it would be a question, because I haven’t heard about a case in which it was argued that the right on the name of the software had also been transferred. But then again, I am not a lawyer and I don’t want to become one…

Jeremiah referred on safety-critical FOSS. Safety critical, he said, was functional safety which means that your device must limp back home at a lower gear if anything goes wrong. He mentioned several standards like IEC 61508, ISO 26262, and others. Some of these standards define “Safety Integrity Levels” which define how likely risks are. Some GNU/Linux systems have gone through that certification process, he said. But I didn’t really understand what copylefted software has to do with it. The automotive industry seems to be an entirely different animal…

If you’ve missed this year’s FOSDEM, you may want to have a look at the recordings. It’s not VoCCC type quality like with the CCCongress, but still good. Also, you can look forward to next year’s FOSDEM! Brussels is nice, although they could improve the weather ;-) See you next year!

Talking on Searchable Encryption at 32C3 in Hamburg, Germany

This year again, I attended the Chaos Communication Congress. It’s a fabulous event. It has become much more popular than a couple of years ago. In fact, it’s so popular, that the tickets (probably ~12000, certainly over 9000) have been sold out a week or so after the sales opened. It’s gotten huge.

This year has been different than the years before. Not only were you able to use your educational leave for visiting the CCCongress, but I was also giving a talk. Together with my colleague Christian Forler, we presented on Searchable Encryption. We had the first slot on the last day. I think that’s pretty much the worst slot in the schedule you could get ;-) Not only because the people are in Zombie mode, but also because you have received all those viruses and bacteria yourself. I was lucky enough, but my colleague was indeed sick on day 4. It’s a common thing to be sick after the CCCongress :-/ Anyway, we have hopefully entertained the crowd with what I consider easy slides, compared to the usual™ Crypto talk. We used a lot imagery and tried to allude to funny stuff. I hope people enjoyed it. If you have seen it, don’t forget to leave feedback! It was hard to decide on the appropriate technical level for the almost 1800 people. The feedback we’ve received so far is mixed, so I guess we’ve hit a good spot. The CCCongress was amazingly organised for speakers. They really did care for us and made sure everything was right. So everything was perfect expect for pdfpc which crashed whenever it was meant to display a certain slide… I used Evince then and it worked…

The days at the CCCongress were intense as you might be able to tell from the Fahrplan. It generally started at about 12:00 and ended at about 01:00. And that’s only the talks. You can’t avoid bumping into VIP (very interesting people) and thus spend time in the hallway. And then you have these amazing parties. This year, they had motor-homes and lasers in the dance hall (last year it was a water cannon…). Very crazy atmosphere. It’s highly recommended to spend a night there.

Anyway, day 1 started for me with the Keynote by Fatuma Musa Afrah. The speaker stretched her time a little, I felt. At the beginning I couldn’t really grasp what her topic was or what she wanted to tell us. She repeatedly told us that we had to “kill the time together” which killed my sympathy to some extent. The conference’s motto was Gated Communities. She encouraged us to find ways to open these gates by educating people and helping them. She said that we have to respect each other irrespective of the skin colour or social status. It was only later that she revealed being refugee who came to Germany. Although she told us that it’s “Newcomers”, not “refugees”. In fact, she grew up in Kenya where she herself was a newcomer. She fled to Kenya, so she fled twice. She told us stories about her arriving and living in Germany. I presume she is breaking open the gates which separate the communities she’s living in, but that’s speculation. In a sense she was connecting her refugee community with our hacker community. So the keynote was interesting for that perspective.

Joanna Rutkowska then talked about trustworthy laptops. The basic idea is to have no state on the laptop itself, i.e. no place where malware could be injected. The state should instead be kept on a personal storage medium, like an SD card or a pen drive. She said that laptops are inherently not trustworthy. Trust, she said can be broken up into Trusted, Secure, and Trustworthy. Secure is resistant to attacks. Trusted is something we, as Security community, do not want to have, like a Trusted third party. Trustworthy, she said, is something different, like the Intel Management Engine which might be resistant to attacks, yet it is not acting in the interest of the user. Application level security is meaningless, she said, when we cannot trust the Operating System, because it is the trusted part. If it is compromised then every effort is not useful. Her project, Qubes OS, attempts to reduce the Trusted Computing Base. What is Operating system to the application, is the hardware to the Operating system. The hardware, she said, has been assumed to be trusted. A single malicious peripheral, like a malicious wifi module, can compromise the whole personal computer, the whole digital life, she said. She was referring to problems with Intel x86 platforms. Present Intel processors integrate everything on the main chip. The motherboard has been made more or less only a holder for the CPU and the memory. The construction of those big chips is completely opaque. We have no control over what is inside. And we cannot look inside, even if we wanted to. The firmware is being loaded during boot from a discrete element on the mainboard. We cannot, however, verify what firmware really is on the chip. One question is how to enforce read-only-ness of the system or how to upload your own firmware. For many years, she and others believed that TPM, TXT, or UEFI secure boot could solve that problem. But all of them have shown to fail horribly, she said. Unfortunately, she didn’t mention how so. So as of today, there is no such thing as a secure boot. Inside the processor is a management engine which special, because it is the perfect entry for backdooring and zombification of personal computing. By zombification, she means that the involvement of the Apps (vs. OS vs. Hardware) is decreasing heavily and make the hardware have much more of a say. She said that Intel wants to make the Hardware fully control your computing by having much more logic in the management engine. The ME is, in a sense, a gated community, because you cannot, whatsoever, inspect it, tinker with it, or otherwise get in touch. She said that the war is lost on X86. Even if we didn’t have the management engine. Again, she didn’t say why. Her proposal is to move all those moving firmware parts out to a trusted storage. It was an interesting perspective on what I think is a “simple” Free Software problem. Because we allow proprietary software, we now have the problem to see what is loaded into the hardware. With Free Software we’d still have backdoors in hardware, but assuming that most functionality is encoded in firmware, we could see and modify the existing firmware and build, run, and share our “better” firmware.

Ilja van Sprundel talked about Windows driver security or rather their attack surface. I’m not necessarily interested in Windows per se, but getting some lower level knowledge sounded intriguing. He gave a more high level overview of what to do and what to not do when doing driver development for Windows. The details, he said, matter. For example whether the IOManager probes a buffer in *that* instance. The Windows kernel is made of several managers, he said. The Windows Driver Model (WDM) is the standard model for how drivers are written. The IO Manager proxies requests from user to (WDM drivers. It may or may not validate arguments. Another central piece in the architecture are IO Request Packets (IRPs). They are being delivered from the IO Manager to the driver and contain all the necessary information for the operation in question. He went through the architecture really fast and it was hard for a kernel newbie like me to follow all the concepts he mentioned. Interestingly though, the IO Manager seems to also care about transferring the correct amount of memory from userspace to kernel space (e.g. makes sure data does not overflow) if you want it to using METHOD_BUFFERED. But, as he said, most of the drivers use METHOD_NEITHER which does not check anything at all and is the endless source of driver bugs. It seems as if KMDF is an alternative framework which makes it harder to have bugs. However, you seem to need to understand the old framework in order to use that new one properly. He then went on to talk about the actual attack surface of drivers. The bugs are either privilege escalation, denial of service, or information leak. He said that you could avoid the problem of integer overflow by using the intsafe library. But you have to use them properly! Most importantly, you need to check their return type and use the actual values you want to have been made safe. During creation of a device, a driver can call either IoCreateDeviceSecure with an SDDL string or use an INF file to ACL the device. That is, however, done either rarely or wrongly, he said. You need to think about who needs to have access to your device. When you work with the IOManager, you need to check whether Irp->MdlAddress is NULL which can happen, he said, if it’s a zero sized buffer. Similarly, when using the safer METHOD_BUFFERED mentioned earlier, Irp->AssociatedIrp.SystemBuffer can also be NULL. So avoid having that false sense of security when using that safe API. Another area of bugs is the cancellation of IRPs. The userland can cancel requests which apparently is not handled gracefully by drivers and leads to deadlocks, memory leaks, race conditions, double frees, and other classes of bugs. When dealing with data from userland, you are supposed to “probe” the memory which is basically checking whether the pointers are valid and in the expected range. If you don’t do that, it’ll lead to you writing to arbitrary kernel memory. If you do validate the data from userspace, make sure you don’t fetch it again from user space assuming that it hasn’t changed. There might be race between your check and your usage (TOCTOU). So better capture, validate, and use the data. The same applies when using MDLs. That, however, is more tricky, because you have a double mapping and you are using a kernel pointer. So it is very subtle. When you do memory allocation you can either use ExAllocatePool or ExAllocatePoolWithQuota. The latter throws an exception instead of returning NULL. Your exception or NULL pointer handling needs to be double checked, he said. It was a very technical talk on Windows which was way out of my comfort zone. I only understood a tiny fraction of what he was presenting. But I liked it for the new insight on Windows drivers and that the same old classes of bug have not died yet.

High up on my list of awaited talks was the talk on train systems by the SCADA strangelove people. Railways, he said, is the biggest system built by mankind. It’s main components are signals and switches. Old switches are operated manually by pure force. Modern switches are interlocked with signals such that the signals display forbidden entry when switches are set in certain positions. On tracks, he said, signals are transmitted over the actual track by supplying them with AC or DC. The locomotive picks up the signals and supplies various systems with them. The Eurostar, they said, has about seven security systems on board, among them a “RPS”, a Reactor Protection System which alludes to nuclear trains… They said that lately the “Bahn Automatisierungssystem (SIBAS)” has been updated to use much more modern and less proprietary soft- and hardware such as VxWorks and x86 with ELF binaries as well as XML over HTTP or SS7. In the threat model they identified, they see several attack vectors. Among them are making someone plug a malicious USB device in controlling machines in some operation center. He showed pictures from supposedly real operation centers. The physical security, he said, is terrible. With close to no access control. In one photograph, he showed a screenshot from a documentary aired on TV which showed credentials sticking on the screen… Even if the security is quite good, like good physical security and formally proven programs, it’s still humans who write the software, he said, so there will be bugs to be exploited. For example, he showed screenshots of when he typed “railway” into Shodan and the result included a good number of railway stations. Another vector is GSM-R. If you jam the train’s GSM-R connection, the train will simply stop. Also, you might be able to inject SIM toolkit malware. Via that vector, you might make the modem identify as, e.g. a keyboard and then penetrate further into the systems. Overall an entertaining talk, but the claims were a bit over the top. So no real train hacking just yet.

The talk on memory corruption by Mathias Payer started off by saying that software is unsafe and insecure. Low level languages trade type safety and memory safety for performance. A large set of legacy applications, he said, are prone to memory vulnerabilities. There are, he continued too many bugs to find and fix manually. So we need a runtime system to ensure security. An invalid dereference or an out of bounds pointer is the core of memory unsafety problems. But according to the C language, he claimed, it’s only a violation if the invalid pointer is read, written to, or freed. So during runtime, there are tons and tons of dangling pointers which is perfectly fine. With such a vulnerability a control-flow attack could be executed. Several defenses exist: Data Execution Prevention prevents code from actually being executed. Address Space Layout Randomisation scrambles the memory locations of executable code which makes it harder to successfully exploit a vulnerable. Stack canaries are special values which are supposed to detect overflowing writes. Safe exception handlers ensure that exception code paths follow predefined patterns. The DEP can only work together with ASLR, he said. If you broke ASLR, you could re-use existing code; as it turns out, people do break ASLR every now and then. Two new mechanisms are Stack Integrity and Code Flow Integrity. Stack Integrity enforces to return to the actual caller by having a shadow stack. He didn’t mention how that actually works, though. I suppose you obtain a more secret stack address somewhere and switch the stack pointer before returning to check whether the return address is still correct. Control Flow Integrity builds a control flow graph during compilation and for every control flow change it checks at run time whether the target address is allowed. Apparently, many CFI implementations exist (eleven were shown). He said they’ve measured those and IFCC and Lockdown performed rather badly. To show how all of the protection mechanisms fail, he presented printf-oriented programming. He said that printf was Turing complete and presented a domain specific language. They have built a brainfuck interpreter with snprintf calls. Another rather technical talk by a good speaker. I remember that I was already impressed last year when he presented on these new defense mechanisms.

DJB and Tanja Lange started their “late night show” by bashing TLS as a “gigantic clusterfuck”. They were presenting on quantum computing and cryptography. They started by mentioning that the D-Wave quantum computer exists, but it’s not useful, he said. It doesn’t do the basic things, and can only do limited computations. It can especially not perform Shor’s algorithm. So there’s no “Shor monster coming”. They recommended the Timeline of Quantum Computing as a good reference of the massive research effort going into quantum computing. If there was a general quantum computer pretty much every public key scheme deployed on the Internet today will be broken. But also symmetric schemes are under attack due to Grover’s algorithm which speeds up brute force algorithms significantly. The solution could be physical crypto like using strong (physical) locks. But, he said, the assumptions of those systems are already broken. While Quantum Key Distribution is secure under certain assumptions, those assumptions are off, he said. Secure schemes that survive the quantum era were the topic of their talk. The first workshop on that workshop happened in 2006 and efforts are still being made, e.g. with EU projects on the topic. The time it takes for a crypto scheme to gain significant traction has been long, so far. They gave ECC as an example. It has been introduced in the 1980s, but it’s only now that it’s taking over the deployed crypto on the Internet. So the time it takes is long. They gave recommendations on what to do to have connections that are secure “for at least the next hundred years”. These include at least 256 bit keys for symmetric encryption. McEliece with binary Goppa codes n=6960 k=5413 t=119. An efficient implementation of such a code based scheme is McBits, she said. Hash based signatures with, e.g. XMSS or SPHINCS-256. All you need for those is a proper hash function. The stuff they recommend for the next 100 years, like the McEliece system, are things from the distant past, she said. He said that Post Quantum Cryptography will be the standard in a couple of years from now so he urged the cryptographers in the audience to “get used to this stuff”.

Next on my list was Markus’ talk on Landesverrat which is the incident of being investigated for revealing secret documents. He referred on the history of the case, how it came around that they were suspected of revealing secret documents. He said that one of their believes is to publish their sources, even the secret ones. They want their work to be critically reviewed and they believe that it is only possible if the readers can inspect the sources. The documents which lead to the criminal investigations were about finances of the introduction of the XKeyscore software. Then, the president of the “state security” filed a case against because of revealing secret documents. They wanted to publish the investigation files, but they couldn’t see them, because they were considered to be more secret than the documents they have already published… From now on, he said, they are prepared for the police raiding their offices, which I suppose is good standard preparation. They were lucky, he said, because their case fell into the regular summer low of news which could make the case become quite popular in the media. A few weeks earlier or later and they were much less popular due to the refugees or Greece. During the press coverage, they had a second battleground where they threw out a Russian television team who entered their offices without having called or otherwise introduced themselves… For the future, he wants to see changes in what is considered to be a state secret. He doesn’t want the government to decide what such a secret is. He also wants to have much more protection for whistle blowers. Freedom of press should also hold for people who do not blog for their “occupation”, but also hobbyists.

Vincent Haupert was then talking on App-based TAN online banking methods. It’s a classic two factor method: Not only username and password, but also a TAN. These TAN methods have since evolved in various ways. He went on to explain the general online banking process: You log in with your credentials, you create a new wire transfer and are then asked to provide a TAN. While ChipTAN would solve many problems, he said, the banking industry seems to want their customers to be able to transfer money everywhere™. So you get to have two “Apps” on your mobile computer. The banking app and a TAN app. However, malware in “official” app stores are a reality, he said. The Google Playstore cannot protect against malware, as a colleague of him demonstrated during his bachelor thesis. This could also been by the “Brain Test” app which roots your device and then loads malware. Anyway, they hijacked the connection from the banking app to modify the recipient of the issued wire transfer and the TAN being pushed on the device. They looked at the apps and found that they “protected” their app with “Promon Shield“. That seems to be a strong obfuscation framework. Their attack involved tricking the root and hooks detection. For the root detection they check on the file system for certain binaries. He could simply change the filenames and was good to go. For the hooks (Xposed) it was pretty much the same with the exception of a few filenames which needed more work. With these modifications they could also “hack” the newer version 1.0.7. Essentially the biggest part of the problem is that the two factors are on one device. If the attacker hijacks that one device then ,

The talk by Christian Schaffner on Quantum Cryptography was introducing the audience to quantum mechanics. He said that a qubit can be imagined as the direction of a polarised photon. If you make the direction of the photons either horizontal or vertical, you can imagine that as representing 0 or 1. He was showing an actual physical experiment with a laser pointer and polarisation filters showing how the red dot of the laser pointer is either filtered or very visible. He also showed how actually measuring the polarisation changes the state of the photons! So yet another filter made the point in the back brighter. That was a bit weird, but that’s quantum mechanics. He showed a quantum random number generator based on that technology. One important concept is the no-cloning theorem which state that you can make a perfect copy of a quantum bit. He also compared current and “post quantum” crypto systems against efficient classical attackers and efficient quantum attackers. AES, SHA, RSA (or discrete logs) will be broken by quantum attacks. Hash-based signatures, McEliece, and lattice-based cryptography he considered to be resistant against quantum based attacks. He also mentioned that Quantum Key Distribution systems will also be against an exhaustive attacker who applies brute force. QKD is based on the no-cloning theorem so an eavesdropper cannot see the same bits as the communicating parties do. Finally, he asked how you could prove that you have been at a certain location to avoid the pizza delivery problem (i.e. to be certain about the place of delivery).

Fefe was talking on privileges. He said that software will be vulnerable. Various techniques should be applied such as simply fixing all the bugs (haha…) or make exploitation harder by applying ASLR or ROP protection. Another idea is to put the program in a straight jacket and withdraw privileges. That sounds a lot like containerisation. Firstly, you can drop your privileges from superuser down to the least privileges you need, then do privilege separation. Another technique is the admin confining the app in a jail instead of the app confining itself. Also, you can implement access control via a broker service by splitting up your process into, say, a left half which opens and reads files and a right half which processes data. When doing privilege separation, the idea is to split up the process into several separately running programs. Jailing is like firewall rules for syscalls which, he said, is impossible for complex programs. He gave Firefox as an example of it being impossible to write a rule set for. The app containing itself is like a werewolf chaining itself to the wall before midnight, he said. You restrict yourself from opening files, creating socket, or from attaching yourself as a debugger to other processes. The broker service is probably like a reference monitor. He went on showing how old-school privilege dropping works. You could do it as easily as seteuid(getuid()), but that’s not enough, because there is the saved UID, so you need to setresuid and not forget to check the return code. Because the call can fail if, for example, the target UID had already been running too many processes for its quota. He said that you should fail the build if your target platform does not provide setresuid. However, dropping privileges is more than setting your UID. It’s also about freeing resources you don’t necessarily need. Common approaches to jailing your process are to have a fake filesystem with only the necessary files, so your process cannot ever access anything that it shouldn’t. On Linux, that would probably be chroot. However, you can escape using fchdir . Also, mounting your /proc into the chroot, information about the host is exposed. So you need to do more work than calling chroot. The BSDs, he said, have Securelevel which is a kernel mode that only increases which withdraws certain privileges. They also have jails which is a chroot on steroids, he said. It leaks some information due the PIDs, though, he said.

The next talk was on Shellphish, an automatic exploitation framework. This is really fascinating stuff. It’s been used for various Capture the Flag contests which are basically about hacking other teams’ software services. In fact, the presenters were coming from the UCSB which is hosting the famous iCtF. They went from solving security challenges to developing a system which solves security challenges. From a vulnerability binary, they (automatically) develop an exploit and a patched binary which is immune to the exploit, but preserves the functionality of the program. They automatically find vulnerabilities, patches, and test both the exploits and the patches. For the automated vulnerability component, they presented Angr. It has a symbolic execution engine looking for memory accesses outside allocated regions and unconstrained instruction pointer which is a jump controlled by user input (JMP eax). They have written a paper for NDSS about “Augmenting Fuzzing Through Selective Symbolic Execution“. Angr is a Python library and they showed how to use it for identifying the overhyped Back to 28 vulnerability. Actually, there is too much state for a regular symbolic executor to find this problem. Angr does “veritesting“. He showed that his Angr script found the vulnerability by him having excluded many paths of execution that don’t really generate new state with a few lines of code. He didn’t show though what the lines of code were and how he determined how the states are not adding any new information.

The next talk was given by the people behind Intelexit was about convincing NSA agents to stop their work and serve democracy instead. They rented a van with big mottoes printed on them, like “Listen to your heart, not to private phone calls”. They also stuck the constitution on the “constitution protection office” which then got torn apart. Another action they did was to fly over the dagger complex and to release flyers about leaving the secret services. They want to have a foundation helping secret service agents to leave their job or to blow the whistle. They also want an anonymous call service where agents can call to talk about their job. I recommend browsing their photos.

Another artsy talk was on a cheap Facebook army. Actually it was on Instagram followers. The presenter is an artist himself and he’d buy Instagram followers for fellow artists “to make them all equal”. He dislikes the fact that society seems to measure the value or quality of art in followers or likes on social media.

Around the CCCongress were also other artsy installations like this one called “machine learning”:

It’s been a fabulous event. I really admire the people organising this event each and every year. Thank you so much and see you next year, hopefully.