Seth Nickell

I get a lot of messages asking me to compare and contrast Storage,
WinFS, and sometimes Dashboard and Medusa. More recently, I’ve gotten a lot of
questions about Spotlight and Beagle. I’ve generally avoided commenting
(which usually means not answering the e-mail…) on these things both because
its impossible for me to do an unbiased comparison, and because the
goals seem to be quite different.

• Medusa, Beagle & Spotlight are similar, though of course Spotlight is
  much more mature. I would call them metadata index systems.
• Storage & WinFS are similar, though of course WinFS is much more mature.
  I would call them document stores.

Caveat: If indexing and search were the
primary goals, a document store would be a ridiculously overengineered
approach. The medusa/beagle/spotlight model is much more sane if
this is your only or primary goal. I’m not saying this to suggest
document stores are better or worse than metadata indexing systems,
only to point out that there’s an element of apple-orange
comparison at work here.

Metadata Index Systems

Medusa:

Medusa was originally written by Eazel integrated tightly with Nautilus
1.0 and was slated for inclusion with the GNOME 1.4 release. It was
primarily written by Rebecca Schulman, but also had major contributions
from Maciej Stachowiak & some by myself. Medusa ran as root, which
worried some people (but of course, so does updatedb for slocate…),
but unfortunately had a major bug that caused it to be pulled from GNOME
1.4 at the last minute. Rebecca fixed the bug after the release, and
re-architected Medusa to run as a normal user. But unfortunately Eazel
collapsed before GNOME 2.0 and nobody promoted its inclusion. Curtis
Hovey & I ported it to GNOME 2.x platform later, and Curtis is currently
maintaining it and adding lots of new features / fixes. In particular he
seems to be working on a UI for it. Medusa allowed very fast searches
over large indexes. Indexes were built by scanning the disk every night
(like slocate, unlike Spotlight which does things better). It also
provided a search: URI scheme that allowed creation of dynamic “search
folders”. So you could have a “Spreadsheets” folder for example that
always contained any spreadsheets on your system. The biggest hurdle for
Medusa today is that the set of indexers is not very extensible, and so
it doesn’t know how to index very many different file types.

Spotlight:

Of course I haven’t looked at Spotlight’s code or used it, so what I
know about it is from what Apple has published and discussions with
friends at Apple. Spotlight appears to be a sophisticated well
implemented approach to building a metadata layer an top of an existing
file system. Changes to files appear to be noticed at the kernel layer,
and indexers are quickly run to update the metadata cache (with
information about filename, album name, size, file contents, keywords,
etc). I don’t know whether it is guaranteed that indexers will be run
before the data can be accessed, but it is supposed to happen very
quickly in any case so it appears instant to the user. Spotlight is the
work of (among others, there are probably more people I just don’t know)
Pavel Cisler (BeOS tracker & Eazel Nautilus) & Dominic Giampaolo (BeOS
BFS, which had a similar sophisticated metadata system). Spotlight also
has a lot of work gone into the UI, for doing grouping, measuring
relevance, etc. Its easy to underestimate how much work this is, in some
ways the “indexing” is the easy part. Spotlight appears to index a lot
more than just the filesystem, including things like calendar and mail,
but I don’t know the full extent of what it can do.

Beagle:

My knowledge of Beagle is based on playing with it and reading through
a fair bit of the code, but I could definitely be missing large aspects
because I haven’t talked with Jon. Beagle’s code appears to be fairly immature at the moment, but I would
expect it to grow. It uses a port of Apache Jarkarta’s Lucene. Lucene
primarily provides a way to *store* indexed metadata and do fast
*searches* over lots of metadata (including full text, of course), but
it doesn’t provide the indexers for specific file types. In some sense,
Lucene as a specialized “database” for storing the results of indexers.
Currently Beagle has indexers for HTML, JPEG, MP3, OpenOffice.org (very
cool) and Text. Unlike Medusa (I have no idea about Spotlight for this)
Beagle is designed to index “byte streams” rather than files, so it can
index, e.g. “The current page you are looking at in Epiphany”. This
makes it very compatible w/ Dashboard, since Dashboard wants to index
any and all contextual data, not just things on the hard disk. At the
moment Beagle appears to contain only very simple UI, so its primarily a
document indexing system.

On the filesystem side, Beagle currently works
like Medusa and requires a “crawler” to update its metadata cache (say
nightly), vs. spotlight which updates instantly. Beagle also has
crawlers for Mail and IM logs. Beagle also includes a renderer system
for displaying the relevant metadata etc for different file type
results. AFAIK, Jon Trowbridge at Novell is the person mainly hacking on
Beagle atm, but I think the code was refactored out of Dashboard, and a
number of other contributors are listed.

Document Stores

Both WinFS & Storage are aimed at doing a lot more than document
indexing… in many ways document indexing is only a nice side effect of
their larger aims. Storage and (AFAICT) to a lesser extent WinFS both
intend to store the actual documents themselves inside the store. That
means that more than just metadata is inside the store. Both WinFS &
Storage provide a query system, though WinFS’ has developed a nice
object oriented language (which I think they compile to SQL) whereas
Storage currently uses straight SQL which is harder for other developers
to use.

Storage:

I know most about this so I’ll talk about it most of course 😉 Storage
is fairly immature, and the architecture has shifted a lot in the past
few months.

“storage-store” provides a DBus service that allows fetching objects
over the FreeDesktop DBus
getting their attributes, relating them to eachother, running queries
etc. “storage-store” uses postgresql to store the structured objects and
perform queries. Because objects are accessed “live” rather than as
“buffers”, changes are instantly propagated across the bus, so multiple
applications or users can work on the same document and instantly see
changes other people make.

I’m currently working on architecture to
storage-store into standard IM presence information so you will be able
to see buddy icons of other people and what part of the document they
are working on inside storage applications. I have a lot of user
experience goals for Storage (or more accurately, for applications and
desktop that use storage). You can find information about most of them
on my blog and at
the storage homepage. Though these goals are more
important to me than document indexing, and have a lot more
impact on Storage’s architecture as a result, I will focus on document
indexing in order to compare and contrast with the other systems.

libstorage-translators provides a framework for translators that can
take structured object data in the store (metadata and the actual data
itself) and translate it to and from byte streams (such as files). The
goal is not indexing files, but for providing a way to move files in and
out of the store. So for example, if your friend sent you a PDF file by
e-mail, you could drag that file into your local store and the
libstorage-translators will automatically decompose the information for
placing in the store (and of course extract lots of metadata like album
name, description, image width, etc etc in the process). Currently I
have only worked on the “importer” side of translators, not the
“exporter”, so they are effectively like indexers. There are currently
importers for: DocBook, HTML, any image format supported by gdk-pixbuf
(JPEG, PNG, BMP, GIF, and several more obscure formats), PDF, text, and
any format supported by gstreamer (MP3, OGG, AVI, MPEG2, etc). Importers
can also create thumbnails for the data for convenient display later.
Storage also includes a renderer system for displaying the relevant
metadata etc for different sorts of results to a query. A major drawback
is that I don’t have translators for common document formats like
Gnumeric or OO.o at the moment.

Queries can either be performed using an SQL-like format (slightly
higher level than SQL but not much, it gets translated to SQL) or using
natural language queries. A large chunk of storage code is currently in
its NL system which uses very sophisticated HPSG grammars and other
techniques to translate human language phrases into the SQL query
format.

A storage:/// VFS URI is provided which automatically invokes
translators when files are dragged into the store. That means you can,
e.g. open a nautilus window to storage:/// and drag files in to add them
to the store. It also provides query folders like Medusa. So for example
you can have a folder “spreadsheets” or “songs by John Lennon that don’t
have the word ‘love’ in them” that is live updated to contain objects
matching those criteria.

WinFS:

I know the least about WinFS of any of the systems
discussed here. I need to read up on it more… but the last time I looked
at it heavily was more than a year ago when MS was still very ellusive.
It looks like a lot of info is up on the web now, so what I’m saying
could be out of date. WinFS is backed by both NTFS & Microsoft’s SQL server.
It provides a very nice API for querying and working with objects.
Currently the set of object types it can used is fixed and predefined by
MS (but the list is long). In the future they will probably open this up
and allow anyone to define new object types. AFAICT, WinFS is currently
targeting primarily the storage of metadata, though it is tightly
coupled to the files themselves stored as byte streams in NTFS. It does
look like in the future they intend to more completely store things in
WinFS. WinFS provides a very cool set of hooks for performing actions in
response to changes in the store. WinFS uses this to provide indexing
services, but users can also define their own actions (e.g. you could
say, “whenever an e-mail from George is created, copy it into my “to
burn” directory”).