Random stuff

Building IoT projects with Ubuntu Core talk

Last week I gave a talk at Perth Linux Users Group about building IoT projects using Ubuntu Core and Snapcraft. The video is now available online. Unfortunately there were some problems with the audio setup leading to some background noise in the video, but it is still intelligible: The slides used in the talk can be found here. The talk was focused on how Ubuntu Core could be used to help with the ongoing security and maintenance of IoT projects. While it might be easy to buy a Raspberry Pi, install Linux and your application, how do you make sure the device remains up to date with security updates? How do you push out updates to your application in a reliable fashion? I outlined a way of deploy a project using Ubuntu Core, including: Packaging a simple web server app using the snapcraft tool. Configuring automatic builds from git, published to the edge channel on the Snap Store. This is also an easy way to get ARM builds for a package, rather than trying to deal with cross compilation tool chains. Using the ubuntu-image command to create an Ubuntu Core image with the application preinstalled I gave a demo booting such an image in a virtual machine. This showed the application up and running ready to use. I also demonstrated how promoting a build from the edge channel to stable on the store would make it available to the system wide automatic update mechanism on the device.

Performing mounts securely on user owned directories

While working on a feature for snapd, we had a need to perform a "secure bind mount". In this context, "secure" meant: The source and/or target of the mount is owned by a less privileged user. User processes will continue to run while we're performing the mount (so solutions that involve suspending all user processes are out). While we can't prevent the user from moving the mount point, they should not be able to trick us into mounting to locations they don't control (e.g. by replacing the path with a symbolic link). The main problem is that the mount system call uses string path names to identify the mount source and target. While we can perform checks on the paths before the mounts, we have no way to guarantee that the paths don't point to another location when we move on to the mount() system call: a classic time of check to time of use race condition. One suggestion was to modify the kernel to add a MS_NOFOLLOW flag to prevent symbolic link attacks. This turns out to be harder than it would appear, since the kernel is documented as ignoring any flags other than MS_BIND and MS_REC when performing a bind mount. So even if a patched kernel also recognised the MS_NOFOLLOW, there would be no way to distinguish its behaviour from an unpatched kernel. Fixing this properly would probably require a new system call, which is a rabbit hole I don't want to dive down. So what can we do using the tools the kernel gives us? The common way to reuse a reference to a file between system calls is the file descriptor. We can securely open a file descriptor for a path using the following algorithm: Break the path into segments, and check that none are empty, ".", or "..". Open the root directory with open("/", O_PATH|O_DIRECTORY). Open the first segment with openat(parent_fd, "segment", O_PATH|O_NOFOLLOW|O_DIRECTORY). Repeat for each of the remaining file descriptors, closing parent descriptors as needed. Now we just need to find a way to use these file descriptors with the mount system call. I came up with two strategies to achieve this. Use the current working directory The first idea I tried was to make use of the fact that the mount system call accepts relative paths. We can use the fchdir system call to change to a directory identified by a file descriptor, and then refer to it as ".". Putting those together, we can perform a secure bind mount as a multi step process: fchdir to the mount source directory. Perform a bind mount from "." to a private stash directory. fchdir to the mount target directory. Perform a bind mount from the private stash directory to ".". Unmount the private stash directory. While this works, it has a few downsides. It requires a third intermediate location to stash the mount. It could interfere with anything else that relies on the working directory. It also only works for directory bind mounts,…

ThinkPad Infrared Camera

One of the options available when configuring the my ThinkPad was an Infrared camera. The main selling point being "Windows Hello" facial recognition based login. While I wasn't planning on keeping Windows on the system, I was curious to see what I could do with it under Linux. Hopefully this is of use to anyone else trying to get it to work. The camera is manufactured by Chicony Electronics (probably a CKFGE03 or similar), and shows up as two USB devices: 04f2:b5ce Integrated Camera 04f2:b5cf Integrated IR Camera Both devices are bound by the uvcvideo driver, showing up as separate video4linux devices. Interestingly, the IR camera seems to be assigned /dev/video0, so generally gets picked by apps in preference to the colour camera. Unfortunately, the image it produces comes up garbled: So it wasn't going to be quite so easy to get things working. Looking at the advertised capture modes, the camera supports Motion-JPEG and YUYV raw mode. So I tried capturing a few JPEG frames with the following GStreamer pipeline: gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=10 ! image/jpeg ! multifilesink location="frame-%02d.jpg" Unlike in raw mode, the red illumination LEDs started flashing when in JPEG mode, which resulted in frames having alternating exposures. Here's one of the better exposures: What is interesting is that the JPEG frames have a different aspect ratio to the raw version: a more normal 640x480 rather than 400x480. So to start, I captured a few raw frames: gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=10 ! "video/x-raw,format=(string)YUY2" ! multifilesink location="frame-%02d.raw" The illumination LEDs stayed on constantly while recording in raw mode. The contents of the raw frames show something strange: 00000000 11 48 30 c1 04 13 44 20 81 04 13 4c 20 41 04 13 |.H0...D ...L A..| 00000010 40 10 41 04 11 40 10 81 04 11 44 00 81 04 12 40 |@.A..@....D....@| 00000020 00 c1 04 11 50 10 81 04 12 4c 10 81 03 11 44 00 |....P....L....D.| 00000030 41 04 10 48 30 01 04 11 40 10 01 04 11 40 10 81 |A..H0...@....@..| ... The advertised YUYV format encodes two pixels in four bytes, so you would expect any repeating patterns to occur at a period of four bytes. But the data in these frames seems to repeat at a period of five bytes. Looking closer it is actually repeating at a period of 10 bits, or four packed values for every five bytes. Furthermore, the 800 byte rows work out to 640 pixels when interpreted as packed 10 bit values (rather than the advertised 400 pixels), which matches the dimensions of the JPEG mode. The following Python code can unpack the 10-bit pixel values: def unpack(data): result = [] for i in range(0, len(data), 5): block = (data[i] | data[i+1] << 8 | data[i+2] << 16 | data[i+3] << 24 | data[i+4] << 32) result.append((block >> 0) & 0x3ff) result.append((block >> 10) & 0x3ff) result.append((block >> 20) & 0x3ff) result.append((block >> 30) & 0x3ff) return result After adjusting the…

Experimenting with C++ Reflection

For a number of projects I've worked on at Canonical have involved using GObject based libraries from C++. To make using these APIs easier and safer, we've developed a few helper utilities. One of the more useful ones was a class (originally by Jussi Pakkenen) that presents an API similar to the C++11 smart pointers but specialised for GObject types we called gobj_ptr. This ensures that objects are unrefed when the smart pointer goes out of scope, and increments the reference count when copying the pointer. However, even with the use of this class I still end up with a bunch of type cast macros littering my code, so I was wondering if there was anything I could do about that. With the current C++ language the answer seems to be "no", since GObject's convention of subclasses as structures whose first member is a value of the parent structure type is at a higher level but then I found a paper describing an extension for C++ Static Reflection. This looked like it could help, but seems to have missed the boat for C++17. However, there is a sample implementation for Clang written by Matus Chochlik, so I downloaded and compiled that to have a play. At its heart, there is a new reflexpr() operation that takes a type as an argument, and returns a type-like "metaobject" that describes the type. For example: using MO = reflexpr(sometype); These metaobjects can then be manipulated using various helpers in the std::meta namespace. For example, std::meta::reflects_same_v<MO1,MO2> will tell you whether two metaobjects represent the same thing. There were a few other useful operations: std::meta::Class<MO> will return true if the metaobject represents a class or struct (which are effectively interchangeable in C++). std::meta::get_data_members_m<MO> will return a metaobject representing the members of a class/struct. From a sequence metaobject, we can determine its length with std::meta::get_size_v<MO>, and retrieve the metaobject elements in the sequence with std::meta::get_element_m<MO> We can get a metaobject representing the type for a data member metaobject with std::meta::get_type_m<MO>. Put all this together, and we've got the building blocks to walk the GObject inheritance hierarchy at compile time. Now rather than spread the reflection magic throughout my code, I used it to declare a templated compile time constant: template <typename Base, typename Derived> constexpr bool is_base_of = ... With this, an expression like is_base_of<GObject, GtkWidget> will evaluate true, while something like is_base_of<GtkContainer, GtkLabel> will evaluate false. As a simple example, this constant could be used to implement an up-cast helper: template <typename Target, typename Source> inline Target* gobj_cast(Source* v) { static_assert(is_base_of<Target, Source>, "Could not verify that target type is a base of source type"); return reinterpret_cast<Target*>(v); } If we can verify that this is a correct up-cast, this function will compile down to a simple type cast. Otherwise, compilation will fail on the static_assert, printing a relatively short and readable error message. The same primitive could be used for other things, such as allowing you to construct a gobj_ptr<T> from an instance of a subclass,…

Extracting BIOS images and tools from ThinkPad update ISOs

With my old ThinkPad, Lenovo provided BIOS updates in the form of Windows executables or ISO images for a bootable CD.  Since I had wiped Windows partition, the first option wasn't an option.  The second option didn't work either, since it expected me to be using the drive in the base I hadn't bought.  Luckily I was able to just copy the needed files out of the ISO image to a USB stick that had been set up to boot DOS. When I got my new ThinkPad, I had hoped to do the same thing but found that the update ISO images appeared to be empty when mounted.  It seems that the update is handled entirely from an El Torito emulated hard disk image (as opposed to using the image only to bootstrap the drivers needed to access the CD). So I needed some way to extract that boot image from the ISO.  After a little reading of the spec, I put together the following Python script that does the trick: import struct import sys SECTOR_SIZE = 2048 def find_image(fp): # el-torito boot record descriptor fp.seek(0x11 * SECTOR_SIZE) data = fp.read(SECTOR_SIZE) assert data[:0x47] == b'\x00CD001\x01EL TORITO SPECIFICATION' + b'\x0' * 41 boot_catalog_sector = struct.unpack('<L', data[0x47:0x4B])[0] # check the validation entry in the catalog fp.seek(boot_catalog_sector * SECTOR_SIZE) data = fp.read(0x20) assert data[0:1] == b'\x01' assert data[0x1e:0x20] == b'\x55\xAA' assert sum(struct.unpack('<16H', data)) % 0x10000 == 0 # Read the initial/default entry data = fp.read(0x20) (bootable, image_type, load_segment, system_type, sector_count, image_sector) = struct.unpack('<BBHBxHL', data[:12]) image_offset = image_sector * SECTOR_SIZE if image_type == 1: # 1.2MB floppy image_size = 1200 * 1024 elif image_type == 2: # 1.44MB floppy image_size = 1440 * 1024 elif image_type == 3: # 2.88MB floppy image_size = 2880 * 1024 elif image_type == 4: # Hard disk image. Read the MBR partition table to locate file system fp.seek(image_offset) data = fp.read(512) # Read the first partition entry (bootable, part_type, part_start, part_size) = struct.unpack_from( '<BxxxBxxxLL', data, 0x1BE) assert bootable == 0x80 # is partition bootable? image_offset += part_start * 512 image_size = part_size * 512 else: raise AssertionError('unhandled image format: %d' % image_type) fp.seek(image_offset) return fp.read(image_size) if __name__ == '__main__': with open(sys.argv[1], 'rb') as iso, open(sys.argv[2], 'wb') as img: img.write(find_image(iso)) It isn't particularly pretty, but does the job and spits out a 32MB FAT disk image when run on the ThinkPad X230 update ISOs. It is then a pretty easy task of copying those files onto the USB stick to run the update as before. Hopefully owners of similar laptops find this useful. There appears to be an EFI executable in there too, so it is possible that the firmware update could be run from the EFI system partition too.  I haven't had the courage to try that though.