Back in December (before I caught the flu working at a farmers market, then Covid two weeks later, then two months of long-Covid) I mentioned that we’d discuss the various subsystems needed in libfoundry to build an IDE as a library.
I used the little bit of energy I had to work on some core abstractions. In an effort to live up to my word lets talk a bit about what went into libfoundry last night.
There is now a DocumentationManager sub-system which handles documentation installed on the host system, chroots, and Flatpak SDKs. It’s a bit tricky to make this all work without blurring the lines of abstraction so lets cover how that works.
Generally speaking, we try to avoid plugins depending on other plugins. Sometimes it happens but usually it is an opportunity to make a better abstraction in libfoundry. Lets look at what is needed around documentation.
- We have many SDKs and they all might have documentation available at different locations.
- We primarily have one format we need to support in GNOME, which is the venerable Devhelp2 XML format serving as an index.
- SDKs might contain the same documentation but at different versions (Nightly vs GNOME 48 vs jhbuild for example)
- There may be more formats that matter in the future especially as we look at pulling in support for new languages.
- Adding new search capabilities shouldn’t break the API.
- Querying needs to be fast enough to update as you type.
So lets dive into the abstractions.
DocumentationManager
This is the core abstraction you start interfacing with. It is a service of the FoundryContext and therefore can be accessed with Foundry.Context:documentation-manager property.
The documentation manager manages the Foundry.DocumentationProvider plug-in abstractions. Plug-ins that which to contribute to the documentation pipeline must subclass this in their plug-in.
To query documentation, use Foundry.DocumentationManager.query(). As I noted earlier, I don’t want new capabilities to break the API so a Foundry.DocumentationQuery object is used rather than a sequence of parameters which would need to be modified.
Avoiding Formats in the API
Since we want to be able to support other documentation formats in the future, it is important that we do not force anything about devhelp2 XML into the core abstraction.
The core result object from queries is a simple Foundry.Documentation object. Like above, we want to avoid breaking API/ABI when new capabilities are added so this object serves as our abstraction to do so. Navigating a tree structure will live here and can be implemented by plug-ins through subclassing.
Additionally, a “devhelp” plug-in provides support for crawling the devhelp2-style directories on disk. But this plug-in knows nothing about where to find documentation as that is relevant only to the SDKs.
This is where the Foundry.DocumentationRoot object becomes useful. SDK plug-ins can implement DocumentationProvider in their plug-in to expose documentation roots. The host-sdk, jhbuild, and Flatpak plug-ins all do this to expose the location of their documentation.
Now the devhelp plug-in can be provided the information it needs for crawling without any knowledge of SDKs.
Fast Querying
The old adage is that the only way to go faster on a computer is to do less work. This is particularly important in search systems where doing an entire query of a database means a lot of wasted CPU, memory, and storage I/O.
To make querying fast the devhelp plug-in indexes information about SDKs in SQLite. Way back in Builder we’d avoid this and just make an optimized fuzzy search index, mmap that, and search it. But now days we’ve gone from one set of documentation to multiple sets of documentation across SDK versions. The problem domain explodes quite a bit. SQLite seemed like a nice way to do this while also allowing us to be lazy in our searching.
By lazy what I mean is that while we’ll start your query, we only retrieve the first few results from the cursor. The rest are lazily fetched as the GListModel is scanned by scrolling. As that is not a very common operation compared to typing, you can throw away a lot of work naturally while still sitting behind the comfortable GListModel interface.
What now?
Since libfoundry already supports SDK management (including Flatpak) you could probably re-implement Manuals in a week-end. Hopefully this also breaks down a bit of the knowledge used to build such an application and the deceptive complexity behind doing it well.
This should also, hopefully soon, allow us to share a documentation implementation across Builder, Manuals, and an upcoming project I have which will benefit from easy access to documentation of object properties.