One of the larger bits of work I’ve been doing crosses many core projects. I’m motivated to switch to GNOME OS across multiple form factors, but to do that, homed and related tooling need quite a few functional gaps fixed.
TL;DR
- github.com/chergert/linux
- github.com/chergert/cryptsetup
- github.com/chergert/qemu
- github.com/chergert/systemd
- github.com/chergert/edk2
I really wanted a rather simple storage stack. I try to run XFS or ext4 in most places because they continue to serve me well. I also want to run GNOME OS on phones, where we’ll need dm-inlinecrypt for better performance and to avoid loading raw storage encryption keys into system memory.
Where I prefer something like thin provisioning is a multi-user setup, where I want encryption, integrity, layering, and reliable accounting without dedicating a fixed partition to every user.
Getting all of those properties at once required work from the filesystem down through device-mapper, the block layer, UFS, Qemu, and cryptsetup.
Keeping the storage key out of memory
Traditional full-disk encryption requires the raw volume key to enter kernel memory. That key can unlock the entire device, so extracting it from a running system is particularly valuable to an attacker.
Hardware-wrapped keys change this arrangement. The long-term key is stored as an opaque, device-bound blob. During activation, it is converted into a boot-scoped ephemeral blob and handed to the storage hardware. The hardware derives and programs the AES-XTS key without disclosing it to software.
There is still a separate 32-byte software secret for integrity and other cryptographic operations which cannot be offloaded. Knowing that secret does not reveal the inline-encryption key.
This reduces the opportunity to extract a reusable storage key, but it is not magic. It does not protect plaintext already present in memory, nor does it defeat an attacker who fully controls the running system.
The practical problem with hardware-wrapped keys is that they are difficult to develop and test without the relevant hardware. Even when hardware is available, failures across the complete stack can be difficult to reproduce and inspect.
So I started building the hardware I needed in Qemu.
A virtual UFS inline-crypto engine
The Qemu work adds an optional UFSHCI 4.1 inline-crypto profile. It supports AES-256-XTS, 512- and 4096-byte data units, 32 keyslots, 64-bit data-unit numbers, and both legacy and MCQ request formats.
It models more than just the encryption operation. Keyslots are programmed in stages before being activated, can be evicted, and are zeroized during reset. Requests take their own snapshots of key state so that concurrent eviction or reprogramming has deterministic behavior. Crypto failures are reported as storage errors rather than returning corrupted data.
There is also a test-only wrapped-key mailbox. It can import or generate a key, prepare a boot-scoped version, derive the associated software secret, program a keyslot, and evict it. The long-term and ephemeral representations use authenticated envelopes so tests can also exercise damaged or substituted blobs.
This mailbox models the API and lifecycle that the guest needs, but it is not a trusted execution environment. Qemu necessarily has access to its root secret.
Following an encrypted write
With the hardware model available, a write can be followed through the entire Linux stack.
dm-inlinecrypt attaches an encryption context to the I/O, including the key and data-unit number. The block inline-crypto layer programs a UFS keyslot, then UFS submits the request to Qemu. Only encrypted bytes reach the backing image. Reads take the reverse path.
This makes dm-inlinecrypt a useful full-device target. Device-mapper describes which blocks should be encrypted, while the actual transform stays in inline-encryption hardware.
The target distinguishes raw keys from hardware-wrapped keys. A long-term wrapped blob is prepared into a fresh ephemeral blob during activation. Only the ephemeral form is placed in an active device-mapper table.
Key replacement also makes secure suspend useful. Userspace can suspend the device and wipe the active key. Resume is refused until a replacement has been installed, at which point a new ephemeral key is prepared and programmed.
Integrity without losing the hierarchy
Encryption by itself does not prevent undetected modification, so the protected configuration places exported dm-integrity above dm-inlinecrypt.
That ordering means integrity authenticates the plaintext seen by the filesystem. Inline encryption protects the filesystem data as well as the integrity tags, superblock, and journal when they reach physical storage.
The HMAC-SHA256 integrity key is derived from the hardware-provided software secret using HKDF-SHA256. The binary LUKS UUID is used as the salt, along with a fixed domain-separation string. This keeps integrity separate from the hardware-only inline-encryption key.
I’ve added a fixed profile at 4096-byte integrity blocks, 32-byte HMAC tags, colocated metadata, and a 32 MiB journal. This is fairly intuition based so it needs more testing.
Suspend and resume follow the layering. Suspend wipes integrity first then inline encryption. Restoration is reversed. This is really hard to test with real hardware, so Qemu again really comes in handy.
Provisioning blocks before publishing them
Thin provisioning adds another problem. A filesystem may publish a logical allocation before the thin pool has assigned physical storage. Failure from lack of capacity is then deferred until too late such as when writing data, an integrity tag, or the integrity journal. All of those can be catastrophic.
I added REQ_OP_PROVISION based on earlier ideas on LKML to make persistent allocation a block-layer op. It is different from a write and is effectively the opposite of discard: it asks the storage stack to ensure that a range is physically backed.
Provisioning is carried through the block core, loop devices, device-mapper, thin volumes, dm-integrity, and ext4. Thin volumes allocate, zero, and commit their mappings. Integrity provisions every corresponding data, metadata, and journal region. Ext4 provisions new data and metadata extents before exposing mappings.
The initial ext4 support is conservative. The provision mount option implies nodelalloc, requires 4 KiB extents without bigalloc, rejects unsupported stacks, and disables online resize.
Turning it into a LUKS2 workflow
The cryptsetup work ties these pieces together.
A platform provisioner can generate or import a wrapped key, derive the optional software secret, format a LUKS2 device for hardware-wrapped encryption, and add ordinary LUKS2 keyslots using the opaque blob as the volume key.
Hardware-wrapped segments have an explicit key_type and mandatory requirements. The integrity configuration is also fixed and marked as dependent on hardware-wrapped-key integrity support. That should make older implementations reject the device.
During activation, cryptsetup retrieves the long-term blob, prepares an ephemeral one, creates dm-inlinecrypt, derives the integrity key, and finally creates dm-integrity above it.
For now, this interface is library-only which is how I’m using it from homed.
Testing all of this in a custom GNOME OS build resulted in me finding some issues in tianocore as well (edk2) which I’ve fixed in my tree to allow booting off PCI-UFS over SCSI.
A laboratory for the whole stack
The important result is that this can now be tested without specialized storage hardware.
We can run AES-XTS known-answer tests and independently inspect ciphertext in Qemu’s backing image. We can test both data-unit sizes, legacy and MCQ queues, fragmented requests, concurrency, reset, cancellation, rekeying, damaged envelopes, and storage errors.
We can also exercise thin-volume provisioning through integrity and inline encryption, reject malformed or downgraded LUKS2 metadata, and verify secure suspend, resume, and key replacement.
Each layer has tests which I tried to keep working and improve along the way.
There is plenty left to do. Namely, I’m not really interested in doing LKML type stuff while unemployed living abroad. So if this is something other people want, they’ll need to encourage their respective teams to pick up the work.
Either way, I now have something useful which is a virtual test lab for a security feature which requires each of these layers to work together.