Talking at GI Tracking Workshop in Darmstadt, Germany

Uh, I almost forgot about blogging about having talked at the GI Tracking Workshop in Darmstadt, Germany. The GI is, literally translated, the “informatics society” and sort of a union of academics in the field of computer science (oh boy, I’ll probably get beaten up for that description). And within that body several working groups exist. And one of these groups working on privacy organised this workshop about tracking on the Web.

I consider “workshop” a bit of a misnomer for this event, because it was mainly talks with a panel at the end. I was an invited panellist for representing the Free Software movement contrasting a guy from affili.net, someone from eTracker.com, a lady from eyeo (the AdBlock Plus people), and professors representing academia. During the panel discussion I tried to focus on Free Software being the only tool to enable the user to exercise control over what data is being sent in order to control tracking. Nobody really disagreed, which made the discussion a bit boring for me. Maybe I should have tried to find another more controversial argument to make people say more interesting things. Then again, it’s probably more the job of the moderator to make the participants discuss heatedly. Anyway, we had a nice hour or so of talking about the future of tracking, not only the Web, but in our lives.

One of the speakers was Lars Konzelmann who works at Saxony’s data protection office. He talked about the legislative nature of data protection issues. The GDPR is, although being almost two years old, a thing now. Several types of EU-wide regulations exist, he said. One is “Regulation” and the other is “Directive”. The GDPR has been designed as a Regulation, because the EU wanted to keep a minimum level of quality across the EU and prevent countries to implement their own legislation with rather lax rules, he said. The GDPR favours “privacy by design” but that has issues, he said, as the usability aspects are severe. Because so far, companies can get the user’s “informed consent” in order to do pretty much anything they want. Although it’s usefulness is limited, he said, because people generally don’t understand what they are consenting to. But with the GDPR, companies should implement privacy by design which will probably obsolete the option for users to simply click “agree”, he said. So things will somehow get harder to agree to. That, in turn, may cause people to be unhappy and feel that they are being patronised and being told what they should do, rather than expressing their free will with a simple click of a button.

Next up was a guy with their solution against tracking in the Web. They sell a little box which you use to surf the Web with, similar to what Pi Hole provides. It’s a Raspberry Pi with a modified (and likely GPL infringing) modification of Raspbian which you plug into your network and use as a gateway. I assume that the device then filters your network traffic to exclude known bad trackers. Anyway, he said that ads are only the tip of the iceberg. Below that is your more private intimate sphere which is being pried on by real time bidding for your screen estate by advertising companies. Why would that be a problem, you ask. And he said that companies apply dynamic pricing depending on your profile and that you might well be interested in knowing that you are being treated worse than other people. Other examples include a worse credit- or health rating depending on where you browse or because your bank knows that you’re a gambler. In fact, micro targeting allows for building up a political profile of yours or to make identity theft much easier. He then went on to explain how Web tracking actually works. He mentioned third party cookies, “social” plugins (think: Like button), advertisement, content providers like Google Maps, Twitter, Youtube, these kind of things, as a means to track you. And that it’s possible to do non invasive customer recognition which does not involve writing anything to the user’s disk, e.g. no cookies. In fact, such a fingerprinting of the users’ browser is the new thing, he said. He probably knows, because he is also in the business of providing a tracker. That’s probably how he knows that “data management providers” (DMP) merge data sets of different trackers to get a more complete picture of the entity behind a tracking code. DMPs enrich their profiles by trading them with other DMPs. In order to match IDs, the tracker sends some code that makes the user’s browser merge the tracking IDs, e.g. make it send all IDs to all the trackers. He wasn’t really advertising his product, but during Q&A he was asked what we can do against that tracking behaviour and then he was forced to praise his product…

Eye/o’s legal counsel Judith Nink then talked about the juristic aspects of blocking advertisements. She explained why people use adblockers in first place. I commented on that before, claiming that using an adblocker improves your security. She did indeed mention privacy and security being reasons for people to run adblockers and explicitly mentionedmalvertising. She said that Jerusalem Post had ads which were actually malware. That in turn caused some stir-up in Germany, because it was coined as attack on German parliament… But other reasons for running and adblocker were data consumption and the speed of loading Web pages, she said. And, of course, the simple annoyance of certain advertisements. She presented some studies which showed that the typical Web site has 50+ or so trackers and that the costs of downloading advertising were significant compared to downloading the actual content. She then showed a statement by Edward Snowden saying that using an ad-blocker was not only a right but is a duty.

Everybody should be running adblock software, if only from a safety perspective

Browser based ad blockers need external filter lists, she said. The discussion then turned towards the legality of blocking ads. I wasn’t aware that it’s a thing that law people discuss. How can it possibly not be legal to control what my client does when being fed a bunch of HTML and JavaScript..? Turns out that it’s more about the entity offering these lists and a program to evaluate them *shrug*. Anyway, ad-blockers use either blocking or hiding of elements, she said where “blocking” is to stop the browser from issuing the request in first place while “hiding” is to issue the request, but to then hide the DOM element. Yeah, law people make exactly this distinction. She then turned to the question of how legal either of these behaviours is. To the non German folks that question may seem silly. And I tend to agree. But apparently, you cannot simply distribute software which modifies a Browser to either block requests or hide DOM elements without getting sued by publishers. Those, she said, argue that gratis content can only be delivered along with ads and that it’s part of the deal with the customer. Like that they also transfer ads along with the actual content. If you think that this is an insane argument, especially in light of the customer not having had the ability to review that deal before loading that page, you’re in good company. She argued, that the simple act of loading a page cannot be a statement of consent, let alone be a deal of some sorts. In order to make it a deal, the publishers would have to show their terms of service first, before showing anything, she said. Anyway, eye/o’s business is to provide those filter lists and a browser plugin to make use of those lists. If you pay them, however, they think twice before blocking your content and make exceptions. That feels a bit mafiaesque and so they were sued for “aggressive geschäftliche Handlung”, an “aggressive commercial behaviour”. I found the history of cases interesting, but I’ll spare the details for the reader here. You can follow that, and other cases, by looking at OLG Koeln 6U149/15.

Next up was Dominik Herrmann to present on PrivacyScore.org, a Web portal for scanning Web sites for security and privacy issues. It is similar to other scanners, he said, but the focus of PrivacyScore is publicity. By making results public, he hopes that a race to the top will occur. Web site operators might feel more inclined to implement certain privacy or security mechanisms if they know that they are the only Web site which doesn’t protect the privacy of their users. Similarly, users might opt to use a Web site providing a more privacy friendly service. With the public portal you can create lists in order to create public benchmarks. I took the liberty to create a list of Free Desktop environments. At the time of creation, GNOME fell behind many others, because the mail server did not implement TLS 1.2. I hope that is being taking as a motivational factor to make things more secure.

Talking at Kieler LinuxTage 2017 in Kiel, Germany

I was invited to present GNOME at the Kieler LinuxTage in Kiel, Germany.

logo

Compared to other events, it’s a tiny happening with something between fifty and hundred people or so. I was presenting on how I think GNOME pushes the envelope regarding making secure operating systems (slides, videos to follow). I was giving three examples of how GNOME achieves its goal of priding a secure OS without compromising on usability. In fact, I claimed that the most successful security solutions must not involve the user. That sounds a bit counter intuitive to people in the infosec world, because we’re trying to protect the user, surely they must be involved in the process. But we better not do that. This is not to say that we shouldn’t allow the user to change preferences regarding how the solutions behave, but rather that it should work without intervention. My talk was fairly good attended, I think, and we had a great discussion. I tend to like the discussion bit better than the actual presentation, because I see it as an indicator for how much the people care. I couldn’t attend many other presentations, because I would only attend the second day. That’s why I couldn’t meet with Jim :-/

But I did watch Benni talking about hosting a secure Web site (slides). He started his show with mentioning DNS which everybody could read, He introduced DNSSEC. Which, funnily enough, everybody also can read, but he failed to mention that. But at least nobody can manipulate the response. Another issue is that you leak information about your host names with negative responses, because you tell the client that there is nothing between a.example.com and b.example.com. He continued with SSH for deploying your Web site and mentioned SSHFP which is a mechanism for authenticating the host key. The same mechanism exists for Web or Mail servers, he said: DANE, DNS-based Authentication of named entities. It works via TLSA records which encode either the certificate or the used public key. Another DNS-based mechanism is relatively young: CAA. It asserts that a certificate for a host name shall be signed by a certain entity. So you can hopefully prevent a CA that you’ve never heard of creating a certificate for your hosts. All of these mechanisms try to make the key exchange in TLS a bit less shady. TLS ensures a secure channel, i.e. confidentiality, non-repudiation, and integrity. That is considered to be generally useful in the Web context. TLS tends to be a bit of a minefield, because of the version and configuration matrix. He recommended to use at least TLS as of version 1.2, to disable compression due to inherent attacks on typical HTTP traffic (CRIME), and to use “perfect forward secrecy” ciphers for protecting the individual connections after the main key leaked. Within TLS you use x509 certificates for authenticating the parties, most importantly in the Web world, the server side. The certificate shall use a long enough RSA key, he said, The certificate shall not use a CN field to indicate the host name, but rather the SAN field. The signatures should be produced with “at least SHA-256”. He then mentioned OCSP because life happens and keys get lost or stolen. However, with regular OSCP the clients expose the host names they visit, he said. Enter OCSP Stapling. In that case the Web server itself gets the OCSP response and hands it over to the client. Of course, this comes with its own challenges. But it may also happen that CAs issue certificates for a host name which doesn’t expect that new certificate. In that case, Certificate transparency becomes useful. It’s composed of three components, he said. Log servers which logs all created certificates, monitors which pull the logs, and auditors which check the logs for host names. Again, your Browser may want to check whether the given certificate is in the CT logs. This opens the same privacy issue as with OCSP and can be somewhat countered with signed log statements from a few trusted log servers.

In any case, TLS is only useful, he said, if you are actually using it. Assuming you had a secure connection once, you can use the TLS Strict Transport Security header. It tells the browser to use TLS in the future. Of course, if you didn’t have that first connection, you can have your webapp entered in the STS Preloading list which is then baked into major browsers. Another mechanism is HTTP Public Key Pinning which is a HTTP header to tell the client which certificates or CAs shall be accepted. The header value is a simple list of hashed certificates. He mentioned the risk of someone hijacking your Web presence with an injected HPKP header. A TLS connection has eventually been established successfully. Now the HTTP layer gets interesting, he said. The Content Type Options header prevents Internet Explorer from snooping content types which might cause an image to be executed as JavaScript. Many Cross-Site Scripting attacks, he said, originate from being embedded in a frame. To prevent that, you can set the X-Frame-Options header. To activate Cross-Site Scripting protection mechanisms, the X-XSS-Protection header can be set. It’s probably turned off by default for compatibility reasons, he said. If you know where exactly your data is coming from, you can make use of a Content Security Policy which is like SELinux for your browser. It’s a bit of a complicated mechanism though. For protecting your Webapp he mentioned Sub-Resource Integrity which is essentially the hash of what script you expect. This prevents tampering with the foreign script, malicious or not.

I think that was one of the better talks in the schedule with many interesting details to be discovered. I enjoyed it a lot. I did not enjoy their Web sites, though, which are close to being unusable. The interface for submitting talks gives you a flashback to the late 90’s. Anyway, it seems to have worked for many years now and hope they will have many years to come.

Talking at PET-CON 2017.2 in Hamburg, Germany

A few weeks ago, I was fortunate enough to talk at the 7th Privacy Enhancing Techniques Conference (PET-CON 2017.2) in Hamburg, Germany. It’s a teeny tiny academic event with a dozen or so experts in the field of privacy.

The talks were quite technical, involving things like machine learning over logs or secure multi-party computation. I talked about how I think that the best technical solution does not necessarily enable the people to be more private, simply because the people might not be able to make use of the tool properly. A concern that’s generally shared in the academic community. Yet, the methodology to create and assess the effectiveness of a design is not very elaborated. I guess we need to invest more brain power into creating models, metrics, and tools for enabling people to do safer computing.

So I’m happy to have gone and to have had the opportunity of discussing the issues I’m seeing. Likewise, I find it very interesting to see where the people are currently headed towards.

Talked at mrmcd 2017 in Darmstadt, Germany

I attended this year’s MRMCD in Darmstadt, Germany. I attended a few times in the past and I think this year’s edition was not as successful as the last ones. The venue changed this year, what probably contributed to some more chaos than usual and hence things not running as smoothly as they did. I assume it will be better next year, when people know how to operate the venue. Although all tickets were sold during the presale phase, it felt smaller than in the last years. In fairness, though, the venue was also bigger this year. The schedule had some interesting talks, but I didn’t really get around to attend many, because I was busy preparing my own shows (yeah, should’ve done that before…).

I had two talks at this conference. The first was on playing the children’s game “battleship” securely (video). That means with cryptography. Lennart and I explained how concepts such as commitment schemes, zero knowledge proofs of knowledge, oblivious transfer, secure multiparty computation and Yao’s protocol can be used to play that game without a trusted third party. The problem, in short, is to a) make sure that the other party’s ships are placed correctly and b) to make sure the other party answers correctly. Of course, if you get hold of the placements of the ships these problems are trivial. But your opponent doesn’t like you to know about the placements. Then a trusted third party would solve that problem trivially. But let’s assume we don’t have such a party. Also, we want to decentralise things, so let’s come up with a solution that involves two players only.

The second problem can be solved with a commitment. A commitment is a statement about a something you’ve chosen but that doesn’t reveal the choice itself nor allows for changing ones mind later. Think of a letter in a closed envelope that you hand over. The receiver doesn’t know what’s written in the letter and the sender cannot change the content anymore. Once the receiver is curious, they can open the envelope. This analogy isn’t the best and I’m sure there’s better real-world concepts to compare to commitment schemes. Anyway, for battleship, you can make the other party commit to the placement of the ships. Then, when the battle starts, you have the other party open the commitment for the field that you’re shooting. You can easily check whether the commitment verifies correctly in order to determine whether you hit a ship or water.

The other problem is the correct placement of the ships, e.g. no ships shall be adjacent, exactly ten ships, exactly one five-field ship, etc. You could easily wait until the end of the game and then check whether everything was placed correctly. But that wouldn’t be (cryptographic) fun. Let’s assume one round of shooting is expensive and you want to make sure to only engage if the other party indeed follows the rules. Now it’s getting a bit crazy, because we need to perform a calculation without learning anything else than “the ships are correctly placed”. That’s a classic zero knowledge problem. And I think it’s best explained with the magic door in a cave.

Even worse, we need to somehow make sure that we cannot change our placement afterwards. There is a brain melting concept of secure multi-party computation which allows you to do exactly that. You can execute a function without knowing what you’re doing. Crazy. I won’t be able to explain how it works in a single blog post and I also don’t intend to, because others are much better in doing that than I could ever be. The gist of the protocol is, that you model your functionality as a Boolean circuit and assign random values to represent “0” or “1” for each wire. You then build the truth table for each gate and replace the values of the table (zeros and ones) with an encryption under both the random value for the first input wire and the random value for the second input wire. The idea now is that the evaluator can only decrypt one value in the truth table given the input keys. There are many more details to care about but eventually you have a series of encrypted, or garbled, gates and you need the relevant keys in order to evaluate it. You can’t tell from the keys you get whether it represents a “0” or a “1”. Hence you can evaluate without knowing the other party’s input.

My other talk was about a probable successor of Return Oriented Programming: Data Oriented Programming (video). In Return Oriented Programming (ROP) and its variants like JOP the aim is to diverge the original control flow in order to make the program execute the attacker’s functionality. This, however, can probably be thwarted by Control Flow Integrity. In its simplest form, it checks on every branch whether it is legit. Think of a database with a list of addresses which are allowed to a list of other addresses. Of course, real-world implementations are more clever. Anyway, let’s assume that we’ll have a hard time exploiting our target with ROP, because we cannot change the CFG of the program. If our attack doesn’t change the CFG, though, we should be safe for anything that detects its modification. That’s the central idea of DOP.

Although I’m not super excited about this year’s edition, I’m looking forward to seeing the next year’s event. I hope it’s going to be a bit more organised; including myself ;-)

Talking at FrOSCon 2017

I attended my first FrOSCon in St. Augusting, Germany. It’s one of the bigger Free Software events in Germany. Supposedly, the Chemnitzer LinuxTage is one of the few events which are bigger than FrOSCon. I thought it’s time for me to attend this event, so I went.

I was scheduled for two talks. One in the very first slot and one in the very last slot. So, to some extent, I was opening and closing the conference ;-) But the official keynote was, to my surprise, performed by Karen. She keynoted the conferences with her “big heart” talk. He told her story about her wanting to find out what software her pacemaker runs. Of course, it was an endless quest with no success. She described herself as a cyborg because of the machinery that is linked up to her body. She researched the security of devices such as pacemakers and found devastating results. In fact, software is deployed in many critical parts with people having no clue how the impact will be if the software is being attacked. She described the honeymoon effect and projected it to the security aspects of deployed software. She described it as a time in which no vulnerabilities are known. But once a vulnerability has been found, the number of known vulnerability increases exponentially. She found a study which shows that Free Software responds better to found vulnerabilities than proprietary systems. She said she went from thinking “Open Source was cool” to “Open Source is essential” because it responds much better in case of security breaches. She cautioned us to be careful with the Internet of Things™, because it will lead to people being connected without the people even knowing. All software has bugs, she said, but with Software Freedom we are able to do something about the situation. It’s been an enjoyable talk and I recommend watching the video.

Another interesting talk was given by Raffa about open data in public transport. Open data, especially in trip planning, can give us better results, he said, because personal preferences can be respected better. But also competition will become tougher if the data is free which might lead to better products. My personal argument in favour of open data is that it would allow offline routing rather than having to connect to the Internet. Some public transportation companies have freed their data, like the companies in Berlin, Ulm, Rhein Neckar, and Rhein Sieg, which is, funnily enough, the local company responsible for the public transport in the area of the event. However, some companies are still hesitant. The reasons are manifold. One is that they don’t want to deal with complaints about wrongly displayed data or simply outdated data that the third party didn’t bother to update. Also abuse is a concern. What would abuse even mean in this context? Well, some companies are afraid that the data is not only being used for trip planning but for finding out how the companies work or what their financial situation is, e.g. by inferring information from the data.

Andreas Schreiber talked about the complications of Open Source in Science. He works at DLR, which is a publicly funded research institute. Software is important to the DLR. 1500 people develop software which costs around 150 Million EUR per year and makes them probably the biggest software house in Germany, he said. As they are producing as releasing software they got in trouble with licensing issues. For example, they released software which was not open source although they thought it was. They also used software themselves which they may not have been entitled to use. Their CIO eventually issued a warning regarding the use and release of Open Source which made the speaker offer workshops and knowledge databases for issues around open source. They created a brochure which they intend to distribute to other institutes, too, because they tend to get more requests for this kind of information from the outside than from the inside of their organisation. I found interesting that the problems, according to the participants of their workshops, are that monetising won’t work, that building a community is hard, and that it costs more time to do “open source” than not which is demotivating. It’s been interesting to learn about issues involved in both consuming and producing open source software.

As I’ve mentioned, I was booked for two events, a talk and a workshop. My workshop was about signing OpenPGP keys. I held a small presentation and ranted, some times a bit unfairly, about the current state of affairs. I showed how people do it as of now and how I think we can do better than that. It’s been the first slot in this conference and the audience was small, albeit larger than expected. We even got to suggest improvements in Gentoo’s packaging, so I consider it a success. My talk (slides) was about how GNOME advances the security of desktop systems. The audience was super engaged and I felt I couldn’t focus so much on other things I only touched upon. But the discussion showed that people do care about a usable desktop. We were talking a lot about dialogues and modal prompts and how they do not contribute to the security of a system. I claimed that they exist because they were cheap for the app developer to do. But we at GNOME, I said, try or at least should try to avoid those as much as possible and we find other ways of enabling the app to capture the user’s intent. I’m surprised that we had such a lively discussion in the very last slot of the conference.

I’m happy to have attended the event and to meet surprisingly many GNOME people! It’s surprisingly close to Frankfurt and Cologne both of which have good connections via plane or train. With around 1800 attendees it’s quite big although the many tracks and rooms make it feel less crowded.

GUADEC 2017

It’s summer and it’s GUADEC time! This year’s GUADEC took place in Manchester, England. It was surprisingly less bad for that location ;-) The organisers deserve a big round of applause for having pulled the event off. After having organised last year’s GUADEC I have first hands experience running such an event. So a big “thank you” to the team from England :)

The venue was a big and modern university and the accommodation was neatly located a few footsteps from the lecture hall. That’s especially nice for the typical English weather ;-) We got to live in the student dorms and I’m a bit jealous of today’s student to be able to live in such a comfortable place.

I attended a few talks from the list, among them was Christian Hergert reporting on The State of Builder which was a bit scattered and not very well structured for beginners like me. I guess was meant to be more of a showing off new features instead of a structured walk through the design and thoughts behind the project. I knew the project existed but I never really got around to work with it so I was a bit put off. But I took that for a good opportunity for installing the latest Flatpaked application :)

I liked Simon’s talk on enabling users to modify the software they are running. Essentially, you can click a button in the application and it’ll fire up an IDE where you can change code and hit “play” to run the new version. Amazing. Software Freedom at its best. He demoed a prototype and I think it’s got potential. I really like the idea of the user being able to tinker around easily. Especially given that the status quo is jhbuild. That’s a nice tool, but it proves to be hard for people to make good use of it. I hope we will see something like this being used in the future.

Federico was telling us about the efforts to make use of the Rust language for GNOME. The gist is, essentially, that you better start with leaf functions of your app or library rather than a central function in your architecture. I then tried to find leaf functions with the help of the compiler, but I failed. I tried Egypt but I wasn’t patient enough to make proper use of the generated dot file in order to identify leaf functions. Maybe I should give cflow a try next time.

I used the BoF days to dip a little bit into Rust. It’s always helpful to have a bunch of smart hackers around. That’s what I like about these kind of events. You get to know and talk to very smart people. I also tried to catch up with my very talented student and discuss the changes we’d like to see.

Thanks to the GNOME Foundation for sponsoring my travel and to the local team for having organised a successful event!

GUADEC 2017 group photo

OpenSuSE Conference 2017 Nuremberg, Germany

As last year I was honoured to be invited to OpenSuSE Conference in Nuremberg, Germany.

The event has grown and I felt a relaxed yet productive atmosphere when entering the venue. Just a few minutes after I arrived I hooked up with interesting people with even more interesting discussions. It was very nice to get together with all the Free Software friends I made over the last years. It was also pleasent to see the event becoming bigger and bigger. I take that as a sign that our community grows although it might also just be consolidation of events.

The organision team provided everything from Brazilian BBQ to perfect weather :) The schedule hosted interesting sessions, including mine of course ;-) I had a small workshop on signing OpenPGP keys and I made people use GNOME Keysign. ;-) It was successful in the sense that we were able to shake out a few bugs to make the application more robust. I also realised that networks are not very permissive nowadays. More precisely, the WiFi blocked mDNS traffic preventing the apps from finding each other :( One design goal of the app was to not have to rely on an Internet connection. But if the networks prevent clients from talking to each other then I think we need to go via the Internet in order to transfer files locally :/ Fortunately, we are working on an Internet transport. Stay tuned for further posts on this issue.

Oh, and we even had a GNOME stand full of amazing stuff.

Talking at GPN 2017 in Karlsruhe, Germany

Although the GPN is an annual event, I haven’t managed to go very often. Last time has already been a while. It’s a pity, because the event is very cute. The location is just amazing which makes being there really fun. It’s a museum hosting many things from our digital world. If you visit only one thing in Karlsruhe, go and visit it. In fact, we tried to organise a small excursion during GUADEC last year. Bloomberg also has an article about the event.

I could only stay one day, but I opened the conference with my talk on building a more secure operating system without sacrificing usability which, of course, was a GNOME related talk. The room was packed and people had to sit on the floor. Based on the feedback I think the people liked to be explained what challenges are to be solved in order to ship more secure systems to more people. You can find the slides here. In case you want to practise your German, you can watch the video here.

The schedule had a few other gems, too. My favourite was Loeschi talking about the upcoming Smart Meter Gateway situation in Germany and how it compares to the rest of Europe. The talk about QR Codes was also nicely done and explained quite well how they work. I hope to be able to attend the event more often :) Especially because I wish the Free Software and the “hacker” people would mingle a bit more.

GNOME Keysign 0.9 released

Oh boy, it’s been a while that we’ve released GNOME Keysign 0.9. We changed quite a few things since last time I’ve reported but the most visible change are the new widgets which I already announced last time. Now it should be much easier to make changes with the GUI and experiment with designs.

Other changes include less visible things like the ability to run the program in a VM. We use gtksink now which not only reduces the amount of code we have to maintain but also makes it easier for us to maintain compatibility with different display servers. Similarly, we don’t use the v4l2src but rather a autovideosrc hoping that it will be more compatible with other platforms.

If you want to try to new version, the instructions in the README should get you going:

pip install --user 'git+https://github.com/GNOME-Keysign/gnome-keysign.git#egg=gnome-keysign'

Alternatively, you may try the Debian or openSuSE package. The Flatpak is still work in progress as we still need to figure out how to work with GnuPG running on the host.

The future will bring exciting changes, too. I plan for i18n support and more Python 3 compatibility.

Installing a “full” disk encrypted Ubuntu 16.04 Hetzner server

I needed to install a server multiple times recently. Fully remotely, via the network. In this case, the machines stood at Hetzner, a relatively large German hoster with competitive prices. Once you buy a machine, they boot it into a rescue image that they deliver via PXE. You can log in and start an installer or do whatever you want with the machine.

The installation itself can be as easy as clicking a button in their Web interface. The manual install with their installer is almost as easily performed. You will get a minimal (Ubuntu) installation with the SSH keys or password of your choice deployed.

While having such an easy way to (re-)install the system is great, I’d rather want to have as much of my data encrypted as possible. I came up with a series of commands to execute in order to have an encrypted system at the end. I have put the “full” in the title in quotes, because I dislike the term “full disk encryption”. Mainly because it makes you believe that the disk will be fully encrypted, but it is not. Currently, you have to leave parts unencrypted in order to decrypt the rest. We probably don’t care so much about the confidentiality there, but we would like the contents of our boot partition to be somewhat integrity protected. Anyway, the following shows how I managed to install an Ubuntu with the root partition on LUKS and RAID.

Note: This procedure will disable your machine from booting on its own, because someone will need to unlock the root partition.

shred --size=1M  /dev/sda* /dev/sdb*
installimage -n bitbox -r yes  -l 1 -p swap:swap:48G,/boot:ext3:1G,/mnt/disk:btrfs:128G,/:btrfs:all  -K /root/.ssh/robot_user_keys   -i /root/.oldroot/nfs/install/../images/Ubuntu-1604-xenial-64-minimal.tar.gz


## For some weird reason, Hetzner puts swap space in the RAID.
#mdadm --stop /dev/md0
#mdadm --remove /dev/md0
#mkswap /dev/sda1
#mkswap /dev/sdb1

mount /dev/md3 /mnt
btrfs subvolume snapshot -r /mnt/ /mnt/@root-initial-snapshot-ro

mkdir /tmp/disk
mount /dev/md2 /tmp/disk
btrfs send /mnt/@root-initial-snapshot-ro | btrfs receive -v /tmp/disk/ 
umount /mnt/

luksformat -t btrfs  /dev/md3 
cryptsetup luksOpen /dev/md3 cryptedmd3

mount /dev/mapper/cryptedmd3  /mnt/

btrfs send /tmp/disk/@root-initial-snapshot-ro | btrfs receive -v /mnt/
btrfs subvolume snapshot /mnt/@root-initial-snapshot-ro /mnt/@

btrfs subvolume create /mnt/@home
btrfs subvolume create /mnt/@var
btrfs subvolume create /mnt/@images


blkid -o export /dev/mapper/cryptedmd3  | grep UUID=
sed -i  's,.*md/3.*,,'   /mnt/@/etc/fstab
echo  /dev/mapper/cryptedmd3   /     btrfs defaults,subvol=@,noatime,compress=lzo  0  0  | tee -a /mnt/@/etc/fstab
echo  /dev/mapper/cryptedmd3   /home btrfs defaults,subvol=@home,compress=lzo,relatime,nodiratime  0  0  | tee -a /mnt/@/etc/fstab

umount /mnt/
mount /dev/mapper/cryptedmd3 -osubvol=@ /mnt/

mount /dev/md1 /mnt/boot

mv /mnt//run/lock /tmp/
chroot-prepare /mnt/; chroot /mnt


passwd

echo  "termcapinfo xterm* ti@:te@" | tee -a /etc/screenrc
sed "s/UMASK[[:space:]]\+022/UMASK   027/" -i /etc/login.defs  
#echo   install floppy /bin/false  | tee -a    /etc/modprobe.d/blacklist
#echo "blacklist floppy" | tee /etc/modprobe.d/blacklist-floppy.conf

# Hrm. for some reason, even with crypttab present, update-initramfs does not include cryptsetup in the initrd except when we force it:
# https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1256730
# echo "export CRYPTSETUP=y" | tee /usr/share/initramfs-tools/conf-hooks.d/forcecryptsetup



echo   cryptedmd3 $(blkid -o export /dev/md3  | grep UUID=) none luks     | tee  -a  /etc/crypttab
# echo   swap   /dev/md0   /dev/urandom   swap,cipher=aes-cbc-essiv:sha256  | tee  -a  /etc/crypttab


apt-get update
apt-get install -y cryptsetup
apt-get install -y busybox dropbear


mkdir -p /etc/initramfs-tools/root/.ssh/
chmod ug=rwX,o=   /etc/initramfs-tools/root/.ssh/
dropbearkey -t rsa -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear

/usr/lib/dropbear/dropbearconvert dropbear openssh \
        /etc/initramfs-tools/root/.ssh/id_rsa.dropbear \
        /etc/initramfs-tools/root/.ssh/id_rsa

dropbearkey -y -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear | \
        grep "^ssh-rsa " > /etc/initramfs-tools/root/.ssh/id_rsa.pub



cat /etc/initramfs-tools/root/.ssh/id_rsa.pub >> /etc/initramfs-tools/root/.ssh/authorized_keys

cat /etc/initramfs-tools/root/.ssh/id_rsa

 
update-initramfs -u -k all
update-grub2

exit

umount -l /mnt
mount /dev/mapper/cryptedmd3 /mnt/
btrfs subvolume snapshot -r /mnt/@ /mnt/@root-after-install

umount -l /mnt

Let’s walk through it.


shred --size=1M /dev/sda* /dev/sdb*

I was under the impression that results are a bit more deterministic if I blow away the partition table before starting. This is probably optional.


installimage -n somehostname -r yes -l 1 -p swap:swap:48G,/boot:ext3:1G,/mnt/disk:btrfs:128G,/:btrfs:all -K /root/.ssh/robot_user_keys -i /root/.oldroot/nfs/install/../images/Ubuntu-1604-xenial-64-minimal.tar.gz

This is Hetzner’s install script. You can look at the script here. It’s setting up some hostname, a level 1 RAID, some partitions (btrfs), and SSH keys. Note that my intention is to use dm-raid here and not btrfs raid, mainly because I trust the former more. Also, last time I checked, btrfs’ raid would not perform well, because it used the PID to determine which disk to hit.



mdadm --stop /dev/md0
mdadm --remove /dev/md0
mkswap /dev/sda1
mkswap /dev/sdb1

If you don’t want your swap space to be in the RAID, remove the array and reformat the partitions. I was told that there are instances in which it makes sense to have a raided swap. I guess it depends on what you want to achieve…



mount /dev/md3 /mnt
btrfs subvolume snapshot -r /mnt/ /mnt/@root-initial-snapshot-ro

mkdir /tmp/disk
mount /dev/md2 /tmp/disk
btrfs send /mnt/@root-initial-snapshot-ro | btrfs receive -v /tmp/disk/
umount /mnt/

Now we first snapshot the freshly installed image not only in case anything breaks and we need to restore, but also we need to copy our data off, set LUKS up, and copy the data back. We could also try some in-place trickery, but it would require more scripts and magic dust.


luksformat -t btrfs /dev/md3
cryptsetup luksOpen /dev/md3 cryptedmd3
mount /dev/mapper/cryptedmd3 /mnt/

Here we set the newly encrypted drive up. Remember your passphrase. You will need it as often as you want to reboot the machine. You could think of using pwgen (or similar) to produce a very very long password and save it encryptedly on a machine that you will use when babysitting the boot of the server.


btrfs send /tmp/disk/@root-initial-snapshot-ro | btrfs receive -v /mnt/
btrfs subvolume snapshot /mnt/@root-initial-snapshot-ro /mnt/@

Do not, I repeat, do NOT use btrfs add because the btrfs driver had a bug. The rescue image may use a fixed driver now, but be warned. Unfortunately, I forgot the details, but it involved the superblock being confused about the number of devices used for the array. I needed to re-set the number of devices before systemd would be happy about booting the machine.


btrfs subvolume create /mnt/@home
btrfs subvolume create /mnt/@var
btrfs subvolume create /mnt/@images

We create some volumes at our discretion. It’s up to you how you want to partition your device. My intention is to be able to backup the home directories without also backing up the system files. The images subvolume might become a non-COW storage for virtual machine images.


blkid -o export /dev/mapper/cryptedmd3 | grep UUID=
sed -i 's,.*md/3.*,,' /mnt/@/etc/fstab
echo /dev/mapper/cryptedmd3 / btrfs defaults,subvol=@,noatime,compress=lzo 0 0 | tee -a /mnt/@/etc/fstab
echo /dev/mapper/cryptedmd3 /home btrfs defaults,subvol=@home,compress=lzo,relatime,nodiratime 0 0 | tee -a /mnt/@/etc/fstab

We need to tell the system where to find our root partition. You should probably use the UUID= notation for identifying the device, but I used the device path here, because I wanted to eliminate a certain class of errors when trying to make it work. Because of the btrfs bug mentioned above I had to find out why systemd wouldn’t mount the root partition. It was a painful process with very little help from debugging or logging output. Anyway, I wanted to make sure that systemd attempts to take exactly that device and not something that might have changed.

Let me state the problem again: The initrd successfully mounted the root partition and handed control over to systemd. Systemd then wanted to ensure that the root partition is mounted. Due to the bug mentioned above it thought the root partition was not ready so it was stuck on mounting the root partition. Despite systemd itself being loaded from that very partition. Very confusing. And I found it surprising to be unable to tell systemd to start openssh as early as possible. There are a few discussions on the Internet but I couldn’t find any satisfying solution. Is it that uncommon to want the emergency mode to spawn an SSHd in order to be able to investigate the situation?


umount /mnt/
mount /dev/mapper/cryptedmd3 -osubvol=@ /mnt/

mount /dev/md1 /mnt/boot

mv /mnt//run/lock /tmp/
chroot-prepare /mnt/; chroot /mnt

Now we mount the actual root partition of our new system and enter its environment. We need to move the /run/lock directory out of the way to make chroot-prepare happy.


passwd

We start by creating a password for the root user, just in case.


echo "termcapinfo xterm* ti@:te@" | tee -a /etc/screenrc
sed "s/UMASK[[:space:]]\+022/UMASK 027/" -i /etc/login.defs
#echo install floppy /bin/false | tee -a /etc/modprobe.d/blacklist
#echo "blacklist floppy" | tee /etc/modprobe.d/blacklist-floppy.conf

Adjust some of the configuration to your liking. I want to be able to scroll in my screen sessions and I think having a more restrictive umask by default is good.


echo "export CRYPTSETUP=y" | tee /usr/share/initramfs-tools/conf-hooks.d/forcecryptsetup

Unless bug 1256730 is resolved, you might want to make sure that mkinitramfs includes everything that is needed in order to decrypt your partition. Please scroll down a little bit to check how to find out whether cryptsetup is indeed in your initramdisk.


echo cryptedmd3 $(blkid -o export /dev/md3 | grep UUID=) none luks | tee -a /etc/crypttab
# echo swap /dev/md0 /dev/urandom swap,cipher=aes-cbc-essiv:sha256 | tee -a /etc/crypttab

In order for the initramdisk to know where to find which devices, we populate /etc/crypttab with the name of our desired mapping, its source, and some options.


apt-get update
apt-get install -y cryptsetup
apt-get install -y busybox dropbear

Now, in order for the boot process to be able to decrypt our encrypted disk, we need to have the cryptsetup package installed. We also install busybox and dropbear to be able to log into the boot process via SSH. The installation should print you some warnings or hints as to how to further proceed in order to be able to decrypt your disk during the boot process. You will probably find some more information in /usr/share/doc/cryptsetup/README.remote.gz.


mkdir -p /etc/initramfs-tools/root/.ssh/
chmod ug=rwX,o= /etc/initramfs-tools/root/.ssh/
dropbearkey -t rsa -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear

/usr/lib/dropbear/dropbearconvert dropbear openssh \
/etc/initramfs-tools/root/.ssh/id_rsa.dropbear \
/etc/initramfs-tools/root/.ssh/id_rsa

dropbearkey -y -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear | \
grep "^ssh-rsa " > /etc/initramfs-tools/root/.ssh/id_rsa.pub

cat /etc/initramfs-tools/root/.ssh/id_rsa.pub >> /etc/initramfs-tools/root/.ssh/authorized_keys

cat /etc/initramfs-tools/root/.ssh/id_rsa

Essentially, we generate a SSH keypair, convert it for use with openssh, leave the public portion in the initramdisk so that we can authenticate, and print out the private part which you better save on the machine that you want to use to unlock the server.


update-initramfs -u -k all
update-grub2

Now we need to regenerate the initramdisk so that it includes all the tools and scripts to be able decrypt the device. We also need to update the boot loader so that includes the necessary Linux parameters for finding the root partition.


exit

umount -l /mnt
mount /dev/mapper/cryptedmd3 /mnt/
btrfs subvolume snapshot -r /mnt/@ /mnt/@root-after-install

umount -l /mnt

we leave the chroot and take a snapshot of the modified system just in case… You might now think about whether you want your boot and swap parition to be in a RAID and act accordingly. Then you can try to reboot your system. You should be able to SSH into the machine with the private key you hopefully saved. Maybe you use a small script like this:


cat ~/.server_boot_secret | ssh -o UserKnownHostsFile=~/.ssh/server-boot.known -i ~/.ssh/id_server_boot root@server "cat - >/lib/cryptsetup/passfifo"

If that does not work so well, double check whether the initramdisk contains everything necessary to decrypt the device. I used something like


zcat /boot/initrd.img-4.4.0-47-generic > /tmp/inird.cpio
mkdir /tmp/initrd
cd /tmp/initrd
cpio -idmv < ../inird.cpio
find . -name '*crypt*'

If there is no cryptsetup binary, something went wrong. Double check the paragraph above about forcing mkinitramfs to include cryptsetup.

With these instructions, I was able to install a new machine with an encrypted root partition within a few minutes. I hope you will be able to as well. Let me know if you think anything needs to be adapted to make it work with more modern version of either Ubuntu or the Hetzner install script.