Redoing File Operation Management in Nautilus

This will serve as a sort of introduction to my project, as well as being a progress update.

Hi, I’m Ernestas and this summer I’m working on Nautilus as part of Google Summer of Code. The goal of the project is to have all I/O operations (i.e. file management, the cache, thumbnailing, searching) managed under a single entity with a capped worker thread count.

This talk by my mentor, Carlos, gives a reason why this is needed in a file manager – searching. Dropping down to the root folder and searching for something as innocent as, say, “a” will likely result in your demise, as Nautilus will happily perform an aggressive benchmark on your system instead of searching. Limiting the number of search jobs will help reduce the random reads performed by the disk as well as leaving the system in a usable state.

Another positive outcome is making the code more object-oriented – going from plain function calls and routing events in a very roundabout way to self-contained task classes with signals to provide progress information (e.g. file task progress, search hits, enumerating directory children for cache purposes) and all other nutrients required for a strong file manager. If the result is not having to split the vim window in four just to work in a nautilus-file-operations.c or nautilus-directory-async.c-equivalent file, I will be damn pleased.

The first two weeks haven’t been very eventful, as the exam schedule in my university clashes a bit with GSoC, but in a few days I will be able to return full-speed. Despite that, one of the goals has been reached – implementing a bare-bones task manager, creating a task API skeleton and porting over a file operation. The wip/ernestask/tasks branch contains the code if you’re interested.

The task manager currently allows limiting the running task count at compile time, but Carlos and I have discussed a more dynamic approach to take, once the groundwork is laid. As the task interface is generic, doing things like preventing conflicting operations as they are queued is a bit of a head-scratcher, but that will probably work out with error callbacks and the like. Another thing that is desirable is turning the manager into a scheduler of sorts, which can assign priorities to tasks, in turn letting short-lived, user-facing tasks take precedence (or something else entirely, this is yet to be decided).

Writing the tasks themselves is an experience in itself, requiring deep and wide class hierarchies (not to mention loads of private helper functions) to accomodate all idiosyncrasies of the code, but so far it’s been relatively straightforward. I’ll bet that a rain jacket will not help in the storm that is going to be shuffling around the code for the cache operations. :)

Talk to you again in a couple of weeks,

Autoconf and Meson walk into a bar…

…and by the time Meson has finished its second drink, Autoconf realizes it’s missing its wallet.

Not a very well-crafted joke, but it serves to make a point.

Building with Autotools is slow.

Although Nautilus has recently gotten better (cutting forty-ish seconds off the configuration phase on my machine, all thanks to Mohammed Sadiq), it takes over a minute to even reach the point where we can begin building, anyway, which is obscene.

I started toying with the idea of switching to Meson in Nautilus completely in late September of 2016 (very convenient, since libgd would soon get a WIP Meson branch). Things were going smoothly, albeit slowly – barely anyone knew what I was doing and there was no pressure to finish it soon. I deemed the work ready in late January/early February (not as convenient, since Meson support would land in libgd master not long after, requiring a tweak on our side) and we were very excited to merge the work – the builds were really fast.

Switching to Meson completely was not going to happen at that point, since it was very late in the cycle and we didn’t want to surprise the packagers near the end. Before deciding that, we had to consider other developers/contributors – not all distributions currently ship the version of Meson we require (>= 0.37.0) – that would mean having to install the latest version through other means (most likely, pip), which might not be an option for some people for personal reasons (I also really like having only distribution packages installed).

With the switch out of the way, we really pushed for its inclusion – rigorous testing (ha) and peer-review happened until midnight, when Carlos ACKed the patch. I pushed it and went to sleep thinking happy thoughts. Things were pretty calm before the storm – issues started cropping up in Continuous. Some of them were things to be fixed in Meson[1][2] (the first thing to completely break the build was printing \u2026 in a message), some were things I had not considered and fixed (unconditional documentation generation!) and some were small mysteries (including the actual source of generated enum descriptions and seeing things build out of order due to very high parallelization).

One of the more subtle breakages, unrelated to Continuous, was projects that build Nautilus extensions. Due to an incompatibility in the generated pkg-config file, those projects would fail to find the version of nautilus-extension they require. The most notable of those being file-roller.

The dust seems to have settled by now and I hope to receive more feedback from developers and contributors as soon as we start defaulting to Meson, but despite all the hiccups along the way, I think we did a pretty good job and hope to see more modules receiving the same treatment. :)

If you’re interested, the rough timeline can be found here (I don’t envy Carlos when he has to review these things).