For a long time I was of the opinion that the problem with file management was the management. And the problem with file systems was the system. If only we could design a better management system we’d be all set. I think that turns out to be wrong. We didn’t need a better horse. We needed a new vehicle.
Everything is a file (until it is better).
This poses quite a dilemma for the design of Finding and Reminding in GNOME 3. So it is worth spending a bit of time investigating.
First, let’s limit the scope to the type of stuff we use files for today. For most people this is basically: Documents, Music, Pictures, Videos, and Downloads.
Documents are a bit hard to define. I guess in the “Places” scheme it more or less meant anything that isn’t Music, Pictures, Videos, or Downloads. An accepted definition seems to be something like: page-based information or evidence usually created by an application. Which probably means that most web pages aren’t documents until saved as PDF or something. This seems pretty close to how we use it.
Clearly, documents have been files. Even Merriam-Webster includes the term file in the definition. But are they always files and will they continue to be? Since quite often Documents are just a shorthand for Office Documents let’s look at what is happening there. Will such documents continue to live solely as files on your computer hard drive and be shared as email attachments? Not with Google Cloud Connect for Microsoft Office. And not with Microsoft’s own cloud initiative. You don’t open a file – you connect to a document.
Even when you look directly at the Google Docs app interface it is really hard to identify the documents as files. It isn’t just a cloud drive with a bunch of files in it. There is no “native” file format. At least to the user, the things just are what they are – spreadsheets, drawings, etc. The unit of storage is an implementation detail that just doesn’t matter until you need to export to another format – which is rare.
And the big news this week is that Apple announced they too will be hosting documents in the cloud:
Every doc, every edit, everywhere.
Not because of hype, not just to make money – but because it is better.
Music is slightly different in that we’re accustomed to both possessing and streaming music. Our purchased music lives in files on our various devices and we have elaborate systems for moving it around and backing it up. However, this too is changing.
Google Music is a serious attempt to dramatically improve the user experience around music consumption. It is:
A better way to play your music. Upload your personal music collection to listen anywhere, keep everything in sync,
and forget the hassle of cables and files.
It isn’t a coincidence that files are mentioned here explicitly.
Yesterday, Apple announced that the key part of iCloud is iTunes in the Cloud. And the pitch for this was in part:
…music you purchase in iTunes appears automatically on all your devices.
Without concerning yourself with downloads or files. Really, iCloud is all about elimination of files from the user experience. That is part of what is meant by “post-PC”.
Amazon also has some interesting things going on here. It is really nice to be able to just click on an album in Amazon’s store and then be able to play it on all of your devices. Without downloading files.
But music not being tied to files isn’t that new. I have playlists on Last.fm, Pandora, and YouTube and at no point does the experience of creating them or using them expose me to files.
Well, you may ask, why do I have so much music on my computer then? Basically, because you put it there and you were working around the RIAA. Once you have a better place to put it you won’t. Music in the cloud is a better user experience.
Pictures are interesting for a few reasons. One thing is that even though you still probably have most of them on your computer you very likely don’t actually interact with the files. Cameras store them in weird ways and there are a boatload of crappy apps that help manage them. Another thing is that you all take a shitload of photos. Which is one reason I suspect that even Apple won’t be permanently storing all photos in iCloud by default. Also, some of you are shy and the rest of you are pervs. đ
In any case, we don’t really think of the content on Facebook, Picasa Web Albums and Flickr as a bunch of files and more than not that is where we are looking at photos.
And again, it isn’t because it is fashionable but because it is better.
Videos can either be similar to photos or similar to music. There really isn’t much different from what I’ve said already in those areas. YouTube, Flickr, Picasa, etc are where we expect to find and share videos.
Downloads are different. In most cases I think this is more for temporary storage for something that doesn’t have a well designed and integrated workflow. In many cases we still need to download software before installing it, download PDFs before printing, or moving to a USB stick or some other type of file bouncing. I hope to investigate this more in future posts.
So, this doesn’t mean that files are going to go away. But we do need to start to think of files differently. Perhaps more like a cache. Perhaps the local system as only one of many possible data providers.
Avoiding files allows us to side step some of the most frustrating things about computer use. When Google says “we want to strip out what frustrates people about operating systems” a large part of what they mean is removing the concept of files. And all their related annoyances: syncing, backups, “I left that at home”, “I’m out of space”, “where did I leave that”, etc. Files are a mechanical world metaphor that retain many of the problems of “things” in the real world.
There really isn’t any disagreement in the industry about this. Where people seem to differ is in how to handle it. I think there are a few approaches here: deep application integration, web primary, and service aggregation. Apple/iOS is doing the first, Google/ChromeOS the second, and maybe HP/WebOS the third.
I think the third approach is the only possible one if you don’t have your own hardware, OS, and huge-ass datacenter.
This all leaves us in Free Software land in a bit of a pickle. Our relevance or even existence depends on how well we can navigate these clouds. But the sky is filled with flak: proprietary licenses, restricted trademarks, limited terms of use, brand “nationalism”, collusion and cooperative agreements.
It is a huge challenge that we must face. Even if you are personally happy to manually manage your local files – the rest of the world isn’t – and this isn’t about you. It is about our future and the future of free culture. This isn’t a problem for tomorrow. This is all happening right now. How do we enable these better experiences without compromising our freedom? What do we do about this new pony? Let’s figure it out together.
Well, youâre so bad and nasty
But I love you, yes I doâ Bob Dylan New Pony
I’m thinking the biggest issue facing the Free community is finding a cloud (or integrating with one? Maybe Google? Ignoring the allegations of them being the new Evil Empire, of the existing clouds, they carry the least baggage, I think). The software side of things really shouldn’t be too difficult with the cloud software that’s starting to show up. We just need a free cloud. The issue is, who will host it? How will it be paid for? Then of course one winds up with all sorts of questions dealing with ethics/morals and copyright. If, say, I rip a CD and host the contents on the cloud, am I violating the law? What if I happen to like being able to look at pictures of my girlfriend in a bikini but don’t want those photos going out to the world at large?
You do not necessarily need a datacenter to store data “in the cloud”. The alternative is storing the data, well, everywhere. That is, using a peer-to-peer network where normal computers are used to store redundant backups, probably with some restriction that you only store (encrypted?) backups for people you know or something like that. Of course, that is not an easy solution, but I think it is the best way to go.Daniel
Jon,
Slightly off-topic… but I don’t know your email address.
Please, please consider “Projects” or multi-“thing” associations as part of your Finding And Reminding work. Better yet, consider Aspects of Projects… source code, documentation, due dates and people participating, etc. These really need to be tied together by the shell in a (hopefully) simple and consistent way. And then history needs to be added to aspects. Woo. Maybe Projects could be auto-linked with a Workspace… open a file in a Project, it pops into the correct workspace.
As a developer, my document history is littered with .c files, makefiles, XML crap and PDF documentation, graphics, icons etc. All of these are carefully placed in a folder structure, but “Recent Documents” is completely useless because of this.
This isn’t developer-specific. I want to be able to start a “Project” (working context in GNOME shell) called “My Wedding” linking guests, photos, web pages, todos, dates, spreadsheets of costs, etc. The Project should not be a container… it shouldn’t be anything other than a name and a context for just about everything else. Deleting a project should only eliminate the context, not the data. It should be easy to split a context into several. Or maybe branch it for experimentation.
I imagine anyone doing something other than simple document editing or web surfing needs some way of tying related stuff together in ways that have nothing to do with file type. Some default Projects (Contexts) could be stuff like Photos.
I have been playing with ideas for how this could be presented in Shell.
Anyway, excuse the brain dump.
For me there are two problems:
a. Application independent sync protocol
We need standard IMAP like protocol(s) for these “files”. Download local copies, offline sync, multiple clients etc. We need this without being tied to a proprietary provider.
How I want to access my music, images and documents is similar to email across different devices, applications and operating systems.
b. organizing them across different providers, different
types of “files”.
I can’t easily organize them. We still can’t tag stuff and easily access them in an application independent way.
Say I had an event with photos, videos, presentations, accounting documents, proposal documents etc. Can I tag them “Event A” and then be able to easily access them? No. So people go back to folders and files.
How would I go about this now in web app world? I go to Flickr, Picasa and some websites to find photos. Google Docs for documents, wait.. somebody has it uploaded on some intranet CMS instead, and the agenda is on another website. Everybody has a copy of the spreadsheet as an attachment in their emails.
You work around this with project management software, intranet systems, personal wiki’s etc.
So as a start, to have the ability to tag stuff from different providers and easily access stuff would be awesome. Ability to tag, and then integrate Tracker + Zeitgest would be an awesome start, even while we sort out the cloud sync and files on different computers issue.
I agree that files are an implementation detail that needn’t be exposed, but I don’t think “cloud computing” is the solution. As long as the “cloud” is really a data centre administered by someone else then the loss (freedom) is always going to outweigh the gain (convenience) in my opinion.
I sincerely hope FreedomBox [http://freedomboxfoundation.org/] takes off, as it could provide some of the benefits of “cloud computing” without sacrificing our freedom.
The cloud document store is a myth. It is only good for grandma/pa doing simple computer work.
People doing serious work do not work on the cloud. This means graphics designers, people who make serious use of office applications (complex excel sheet, etc…), Audio Work, Developers, Engineers, etc…
They might use a dropbox like cloud storage to back up, but definatly not as the main way to manage their documents.
Thanks for sharing your ideas. I’d like you to keep considering the humongous part of the world population without good enough internet access to make anything cloud based meaningful. Several of the people I work with still need to leave their homes to get internet access. 3G type services are also very expensive for us and prohibits work on anything except small files and short times.
I’m from South Africa, and I guess our circumstances are better then many other places. Please don’t lock us out by assuming the cloud is everything. It is cool, maybe better in some ways, but for practical purposes it doesn’t even exist for some people.
Not everything belongs in the cloud.
Keeping files local also has advantages. First of all control.
And thatâs one of the reasons why we use free software, right? To keep control of things and not make some company decide and save things, potentially exposing security holes.
So do we need free software for cloud services?
Still a yes.
[WORDPRESS HASHCASH] The poster sent us ‘0 which is not a hashcash value.
Some excellent points. I’d just like to throw out my standard warning that it’s very easy for people living in cities in first-world countries to assume that the internet is always-on and we can completely abstract away the connection. Pushing things to the cloud is great, I love my dropbox (I’ll love it more when I move to a libre equivalent), but we need to remember that some computers never get connected to the internet in their whole lives.
Good points. On a totally unrelated topic: is that you with the striped t-shirt in the photo? đ
Wow… Somebody has been drinking the Apple Kool-Aid. I might sound old-fashioned here (like the 27-year old geezer that I am), but have you ever thought about the fact that people use free software precisely *because* we don’t do all this “lock my stuff up in your cloud” crap? I mean, having certain things synced and abstracted on a mobile device is all fine and dandy, but I like to get actual work done on my desktop. Even without a net connection. I produce content and for that I need to know where my *files* are.
Why should I sync my music library to the cloud? I run a very modest all-SSD Thinkpad and still every song Bob Dylan, Neil Young, John Fogerty and Bruce Springsteen ever wrote all fit on my disk about 100 times over. I don’t need to iSync my iMusic to the iCloud… I have it right HERE! I think what you don’t understand is that “the industry” is adopting many of these solutions not because it is better or more convenient for you, but because it gives them control over you and enables them to easier sell you stuff you could just as well do at home for free.
If free software has one chance at conquering the consumer market then it’s the fact that we aren’t ruled by corporations that need to place their desire to make money above everything else. If we aren’t so stupid to copy everything these corporations do because we believe their PR bullcrap, that is an immense advantage that gives us a lot of power. Or simply said: Just because Apple is dumbing their OS down to appeal to the iPad crowd, we shouldn’t do the same. We need to be different from these solutions to succeed. The fact that we don’t have huge server farms to sync everyones stuff and tie them up with DRM is an *advantage* not a regretable problem. Let us be the option for intelligent users that like to create instead of mindlessly consuming what other people want them to consume. Let us create federated, free, superior solutions and UIs for intelligent users. The kind of users that Apple is *supposedly* targeting…
Not saying that Jobs is the devil, but I think Dylan nailed what “the industry” is trying to do to its consumers perfectly in “Man of Peace”:
He got a sweet gift of gab, he got a harmonious tongue
He knows every song of love that ever has been sung
Good intentions can be evil
Both hands can be full of grease
“But the sky is filled with flak: proprietary licenses, restricted trademarks, limited terms of use, brand ânationalismâ, collusion and cooperative agreements.”
Currently, each provider can have their own go at this because there’s no standard. Kickstart a vendor-neutral ecosystem; if there are enough users willing to pay, hosting providers will show up. It doesn’t have to solve all the problems you mention from the outset, there just has to be one provider and enough incentive for the first users to show up.
I wouldn’t mind paying a couple of USD per month if it would free me from worries about backups and automatic syncs, and was integrated with my desktop of choice, AND if I could switch provider easily.
PS: Your blog font is small and really low on contrast. Humbly suggest you bump it up a couple of notches.
@John Dulaney:
> We just need a free cloud. The issue is, who will host it?
You will, or whoever else you pick. IMHO âyour own cloudâ is much more important issue than how to make UI even more useless^H^H^H^H^H^Hstreamlined. I think the main FLOSS community issue in the next years is how to take all the initiatives for own clouds (Unhosted, Freedombox, etc.) and its components (Zimbra, Zarafa, Davical, etc.) and make it all into one nice (or more) package which would even less sophisticated user can run on his own server, or which could be hosted by the independent companies.
Something that would be good is free software to manage these types of data, and many other types as well. Software that those who want can install on their own servers, and others can use paid or ad supported public services.
That way you can have your privacy, by limiting who has access to what, and controlling your own server. At the same time, you can avoid the complications with syncing data between devices, and you can share links with other people.
Very good post. I also think that the Finding and Reminding effort should aim to replace the “files and folders” experience with something better instead of just finding workarrounds for the problems of dealing with files and folders. Of course, files and directories are an important implementation, but the experience needn’t mirror the implementation.
Those could storage systems are examples of “post-file” experience, but it doesn’t mean that the cloud is a requirement for that user experience. I dream of a Goodle Docs-like post-“files and folders” experience, but dealing with data stored in my computer instead. Or, even better, dealing with all my data, regardless of its local or remote location. That would cover the “I don’t want to store this thing in the cloud” and “I usually work off-line (lack of Internet connection)” use cases.
I think we have a bigger challenge. The real benefit from the cloud is that folks like Microsoft and Google can monetize it. All the other so-called advantages are really marketing hype. Those that want to monetize it will make sure it is as proprietary as possible.
The cloud could turn into an existential challenge for open software. Certainly the Microsofts and Apples of the world will be working hard to make that happen. The challenge for us isn’t just getting something similar and free. To be successful, we are going to have to be better — a lot better.
The filesystem should no longer be directly exposed to the end user.
From a technical perspective, the filesystem is part of the OS and has to do specific things. From an end user’s perspective, what a filesystem does is “legacy” and not necessarily how they want to think about their work.
Just as applications are “icons” to users and a whole bunch of complex .so, .rpms to techies, so “data” needs to be abstracted away from the technical “file in a folder” implementation.
Excellent conversation.
I agree that file management based on cloud or local storage needs to be an option. We all have files that need to be shared and used amongst many computers and other files that can not or should not be given to a third party.
I and many other corporate slugs are under several NDA’s (Non Disclosure Agreements) which forbid us from sharing documentation or designs with third party vendors. I have worked as a consultant in the content management industry for more years than I care to share openly. The legal depart is usualy our main client. One of the first things we do is find out how many iPads and cloud based devices are present in the organization so legal can shut them down.
The cloud is great for personal and open source files, but is still considered a legal liability for people working in the corporate world.
For users, “file” is really the same thing you call “document”; except for files the user doesn’t/shouldn’t see (e.g. /usr/, ~/.local), a “document” is hard to define precisely because the details are not anticipated.
The “file” abstraction is necessary because it doesn’t make _any_ sense for each application to do “file management” – e.g. worrying about backups, about data transfer using USB sticks, emails, bittorrent, about encryption and access control. Implementing these things into each application separately just doesn’t make any sense, and an OS should provide them – and that’s what the “file” concept really is.
Even in the “cloud” environment, implementing these things into each web application (Google apps / MS Office Live) does not make much sense – the user “deserves” a consistent approach and UI.
If the interface has a web browser, you have a thin client.
If the interface supports contacts, emails, photos and videos, you have a appliance that can store contacts and display/edit photos and videos.
If the interface supports files, you have a general-purpose computer.
“Users do not need to manage files” is equivalent to “we are building a thin client/appliance, not a general-purpose computer”.
Agreed that normal users shouldn’t necessarily have to deal with files; it really makes sense to present to them some more high-level “thing” which includes documents, pictures, music, mails, bookmarks, contacts, events, but excludes stuff like /etc/hosts and ~/.history (which are files but not user-visible “things”).
Also agreed that the cloud stuff which is coming up now might be nice to use, and might be beneficial to the end user (actually I use something like this for some years now, namely an IMAP server which allows to read and sort my private mail from home and from work, and via web interface actually from anywhere. I wouldn’t want to miss this flexibility).
So much for the goals, and on to the “where are we now”: actually, the “file” is quite a success story. It is a de-facto standard which allows to exchange data between different operating systems, hardware architectures, device classes, and services. Nearly any computer-like system nowadays supports the idea of “files” (I can think of some embedded microcontrollers which do not support this, but I think this is already a dwindling group). So if there ever comes up a new “data exchange system”, maybe one should first analyze what made the “file” concept that successful.
Regarding the cloud thing: I wouldn’t want to store my data exclusively in Googles cloud system. First of all, I want to retain control about who accesses my data; second, I don’t want to rely on Google and on internet connections to access this data myself. So rather than designing a system of “cloud storage + transparent local caching”, shoot for a system of “local storage + easy transparent data transfer”. That way, the cloud is no longer a pillar of the data storage concept (but rather an optional addition); and unreliable internet connections are no disaster (but rather a temporary annoyance occurrence).
So in conclusion, if the FOSS community wants to take part of the cloud stuff, maybe it should look at its previous factors to success: the pervasive file standard, and reliance on many independent minds rather than one large contributor.
Has anyone thought of making a decentralised “cloud” for Gnome/Linux?
Rather than a Gnome server that hosts all my files, how about distributing them encrypted over the machines of all the other gnome users?
Bit like what they’re doing with that Dispora Facebook killer …
Urrgh.
I’m fine with improving on the notion of a filesystem. Really. Arranging files is hard; sometimes I want mutually-exclusive hierarchical classifications, sometimes I want tags, sometimes I want full-text search, sometimes I don’t. Sometimes files live other places, sometimes files aren’t mine, sometimes files are really copies of remote stuff that I’d like to find again so I can update or share it. These would be great problems to solve.
But I’m very, very apprehensive that the text of this post seems to consider Windows XP as the pinnacle of document arrangement. My documents are more interesting than one of four MIME prefixes, and for that matter, so am I. So while I encourage thinking along these lines, please please don’t fall into the engineering trap of stuffing the world into four buckets and calling it a day. We had that ten years ago. It sucked.
I don’t yet see how The Cloud is strictly better, either. It’s been, what, five years at least since the end of the desktop was heralded? What do we have to show for it? Google Docs is trotted out as the example every time, and I can’t stand the thing. I have no documents I created myself, just ones other people shared with meâhalf the point of Google Docs, rememberâand yet the homepage is sorted by modification date. As far as I can tell, sorting or filtering by who made it is impossible. I can put things in collections and see others’ collections, I guess, but I can only do that from within a document… so Google Docs is no better than a directory on my computer, except it’s much harder to organize, and all the documents are trapped within Google Docs. The future looks bright.
No one building these systems seems to have any idea how to actually improve on what we have. The new approaches don’t have any more context than dumb filesystems, really. Off the top of my head:
– Full-text search has been unhelpful noise from Windows XP’s introduction all the way through Thunderbird 3; I’ve always had to turn it off to actually find what I want.
– You, and every other desktop designer, say “Pictures” as though it were a synonym for “photos I took”. What about photos other people took that I saved? Wallpaper? Diagrams and references? Sprites? Rendered art made for me? Rendered art I made? Videos have similar problems; my own camera shots, trailers, movies, music videos, etc. are fundamentally different kinds of things.
– For that matter, what am I supposed to do with anything I create myself? The notes, the half-finished product, the abandoned forks, the various bits and pieces that aren’t useful by themselves. Where do these go?
– I’ve been trying out Synapse, which I think is built upon Zeitgeist. It keeps suggesting recently-played videos to me. Those seem intuitively like that last ones I’d want to see, since… you know… I just saw them.
dbrodie kinda nailed this: every fancy new content management system seems geared towards a smiling middle-aged father in a Best Buy ad, who uses his computer for two rolls of vacation photos and emailing his kids in college. This is a huge disservice to most users; even my middle-aged father has been doing home video editing and TV recording for decades, and no system has been able to handle that any better than “well I guess these go in My Videos”.
The filesystem is flexibleâperhaps not infinitely so, but well enough that it can handle generic uses well enough. Dumping all my content in one of four buckets, and letting me sort it by mtime, is not a sufficient replacement. It’s the same approach that left GNOME Shell barely usable for me; some engineers came up with a “simpler” paradigm that they think works most of the time, so now I have to futz around when I want to run gvim a second time because it never occurred to anyone that I might want to look at two documents at once.
[WORDPRESS HASHCASH] The poster sent us ‘0 which is not a hashcash value.
Let the user set up his own Cloud, on a own box or at his main computer. The user should have a place to set up and find his own ‘cloud’ on other devices.
This would be feasible for computer-computer, but computer-mobile would be more of a hassle. That’s where Apple got the stronger hand, because of their tightly nit ecosystem.