Doing Things That Scale

There was a point in my life when I ran Arch, had an elaborate personalized terminal prompt, and my own custom icon theme. I stopped doing all these things at various points for different reasons, but underlying them all is a general feeling that it’s taken me some time to figure out how to articulate: I no longer want to invest time in things that don’t scale.

What I mean by that in particular is things that

  1. Only fix a problem for myself (and maybe a small group of others)
  2. Have to be maintained in perpetuity (by me)

Not only is it highly wasteful for me to come up with a custom solution to every problem, but in most cases those solutions would be worse than ones developed in collaboration with others. It also means nobody will help maintain these solutions in the long run, so I’ll be stuck with extra work, forever.

Conversely, things that scale

  1. Fix the problem in way that will just work™ for most people, most of the time
  2. Are developed, used, and maintained by a wider community

A few examples:

I used to have an Arch GNU/Linux setup with tons of tweaks and customizations. These days I just run vanilla Fedora. It’s not perfect, but for actually getting things done it’s way better than what I had before. I’m also much happier knowing that if something goes seriously wrong I can reinstall and get to a usable system in half an hour, as opposed to several hours of tedious work for setting up Arch. Plus, this is a setup I can install for friends and relatives, because it does a decent job at getting people to update when I’m not around.

Until relatively recently I always set a custom monospace font in my editor and terminal when setting up a new machine. At some point I realized that I wouldn’t have to do that if the default was nicer, so I just opened an issue. A discussion ensued, a better default was agreed upon, and voilà — my problem was solved. One less thing to do after every install. And of course, everyone else now gets a nicer default font too!

I also used to use ZSH with a configuration framework and various plugins to get autocompletion, git status, a fancy prompt etc. A few years ago I switched to fish. It gives me most of what I used to get from my custom ZSH thing, but it does so out of the box, no configuration needed. Of course ideally we’d have all of these things in the default shell so everyone gets these features for free, but that’s hard to do unfortunately (if you’re interested in making it happen I’d love to talk!).

Years ago I used to maintain my own extension set to the Faenza icon theme, because Faenza didn’t cover every app I was using. Eventually I realized that trying to draw a consistent icon for every single third party app was impossible. The more icons I added, the more those few apps that didn’t have custom icons stuck out. Nowadays when I see an app with a poor icon I file an issue asking if the developer would like help with a nicer one. This has worked out great in most cases, and now I probably have more consistent app icons on my system than back when I used a custom theme. And of course, everyone gets to enjoy the nicer icons, not only me.

Some other things that don’t scale (in no particular order):

  • Separate home partition
  • Dotfiles
  • Non-trivial downstream patches
  • Manual tracker/cookie/Javascript blocking (I use uMatrix, which is already a lot nicer than NoScript, but still a pretty terrible experience)
  • Multiple Firefox profiles
  • User styles on websites
  • Running your blog on a static site generator
  • Manual backups
  • Encrypted email
  • Hosting your own email (and self-hosting more generally)
  • Google-free Android (I use Lineage on a Pixel 1, it’s a miserable existence)
  • Buying a Windows computer and installing GNU/Linux
  • Auto-starting apps
  • Custom keyboard shortcuts, e.g. for launching apps (I still have a few of these, mostly because of muscle memory)

The free software community tends to celebrate custom, hacky solutions to problems as something positive (“It’s so flexible!”), even when these hacks are only necessary because things are broken by default. It’s nice that people with a lot of time and technical skills can fix their own problems, but the benefits from that don’t automatically trickle down to everybody else.

If we want ethical technology to become accessible to more people, we need to invest our (very limited) time and energy in solutions that scale. This means good defaults instead of endless customization, apps instead of scripts, “it just works” instead of “read the fucking manual”. The extra effort to make proper solutions that work for everyone, rather than hacks just for ourselves can seem daunting, but is always worth it in the long run. Just as with accessibility and commenting your code, the person most likely to benefit from it is you, in the future.

25 thoughts on “Doing Things That Scale”

  1. I simply wholeheartedly agree with everything you said here. It’s cool to DIY sometimes, but not all the time, and especially not for something that you’ll have to maintain every day in the future.

  2. I understand what you mean. Basically, don’t use workarounds, fix it in first place. And leaving the default path could mean pain, if not tested well.

    On the other side. I’m using for years Fedora and Arch. Installing Fedora is quick, while the Installer is still weird. Keeping an Arch running is – well – just keep it running and don’t install ever again. Arch does what Linux is supposed to do, working for me reliable.

    I‘m sadly rather good in killing Windows by bad handling (myself) or bad luck (lose power). When something goes wrong during the Windows-Update shutdown/reboot it is almost always unrecoverable dead.

    Regarding Fedora. Use SWAP and don’t use XFS. Despite you have a enough RAM and XFS[1] is reliable (it is the RHEL default). I suspect to have broken my Fedora twice during major upgrades, because it still relies on SWAP or something awkward. The XFS/Plymouth was also terrible.

    [1]https://fedoraproject.org/wiki/Common_F27_bugs#GRUB_update_might_fail_if_.2Fboot_is_on_XFS

    1. Hi,

      Congratulations for understanding the benefits of scaling. However, the example you give falls short of the mark.

      First, DejaVu Sans Mono is not an “horrible” font. It was nice enough to be forked with minimal changes by Apple (as “Menlo”). So it has all the necessary designer creds, even if you disagree with them.

      Font design is very subjective, and studies show that what people call “taste” is mainly driven by habits, because the brain share recognition routines self-train, and feels good about shapes it trained against.

      There are very few free software monospace fonts on the market that match DejaVu Sans Mono Unicode coverage. Unicode coverage matters because terminal and editor users do far more than just displaying ASCII text on in their monospace apps. Better defaults may possibly be defined today but they need to be argued a lot better than in the GNOME ticket.

      Second, doing things that scale means fixing things at the correct technical level. Which GNOME isn’t. If you disagree with fontconfig default settings, make the change happen fontconfig-side, do not add an app-level override (vetted by people with little understanding of the technical constrains, since they are not text stack maintainers).

      Right now all you’ve made happen is an override that will interfere with further maintenance and enhancement of the common text stack. Which is ultimately detrimental for all text stack users (starting with GNOME). Doing things at the wrong level because you know one project better than others is textbook not scaling, please don’t do that.

      Scaling means making change happen as far upstream as you can get, if you don’t want to do the upstream dance, keep things as local tweaks, that will do less damage.

      Changes *can* be made downstream, but only after trying to make them happen upstream, when upstream clearly does not care about a problem.

      1. > Second, doing things that scale means fixing things at the correct technical level. Which GNOME isn’t.

        Why is GNOME (i.e. the platform level) the wrong level? Default fonts are very much a platform thing, precisely because it’s a somewhat subjective thing. Are you suggesting every platform should use the same default fonts?

        > Right now all you’ve made happen is an override that will interfere with further maintenance and enhancement of the common text stack.

        How so?

    1. If the problem you’re solving is “I want to have a website to post my articles on”, then I think the solution should probably not involve git, local builds from the terminal, or CNAME configs to get a custom domain.

      Static site generators are cool conceptually, but in order to make them scalable there’d need to be a GUI app or something that does all the boring build/deploy stuff for you.

      1. Coworkers are working on ColdCMS, which is a WYSIWYG CMS (Wagtail, based on Django) all packaged up in a way that changes (writing a new page, etc…) generate a static website.

        This way, visitors get a super fast website (no DB access, …) and you still get a powerful and accessible CMS instead of an elitist setup based on manually editing markdown and pushing to got so that a hook rebuilds/redeploys your website. (That’s the setup for my own website )

        Hit me up if you’re interested and want to talk about it.

        1. That’s very interesting indeed! I’ve sent you an email about it. For what I’ve found myself, it seems that Netlify CMS is the kind of tool Tobias is talking about, but maybe it still leans more on the elitist side.

      2. Github pages + Jekyll? Software Carpentry uses that quite effectively. Additionally, unless you plan on using a hosted instance of whichever CMS you’ve chosen (which likely means some level of lock in), you’re going to need someone to maintain the system it runs on. Neocities is another option, which comes with tutorials.

    1. Well it’s kinda already in there. Buying Windows computers and installing GNU/Linux definitely does not scale.

      If we want this software freedom thing to work out we need to actually make complete products, which means hardware + software + services.

      1. My comment was tongue in cheek but yet your argument might be a slippery slope, although I fundamentally agree with you.

        1. I know your comment was not serious, however, Tobias’s response treated it seriously. There is a big difference. With Windows, you can’t get upstream to fix things. You are stuck with upstream’s defaults, same as with closed source apps. You can ask upstream to make changes; that’s the best outcome.

          With Linux, you can fix things. Tobias’s choice is to go to the effort of upstreaming the fix (which may involve some compromise), or not.

          I don’t understand the relevant of Linux not scaling due to laptop installs. I suppose if you have a fleet of laptops. I haven’t managed a laptop fleet of size > 5, but the cost of maintaining Linux/Windows/macos over the lifetime of the machine is much more effort than the initial install, and I think Windows is the worst of the three, so it’s hard for me to see how the cost of the initial install remains very significant.

  3. Great blog ! I completely agree and I’ve been through a similar process … and am still stuck with others, like using uMatrix .

    However, we should be honest that some people just have crazy personal preferences, and we need to be careful about creating an idea that “you can just make your personal preference become the default for everyone”. I know that isn’t what you said in this blog, but it only takes one lazy reader to turn it into that.

  4. Definitely, to build a cross-architecture ecosystem will be necessary hardwares (phones, tablets, laptos, desktops, why not car and media centers?), softwares (system, apps, games) and services (cloud storage and sync) that Scale.

  5. I agree with pretty much everything, except “having /home as a separate partition” being a non-scalable thing. A separate /home has no cost, only benefits. If there’s _one_ thing that definitely scales and lets me be efficient when it comes to managing machines, it’s that one.

    It lets you clean-install a Linux distro and be “up and running” in 20 minutes flat (OK, plus the time to copy-paste your big dnf command of extra packages to install… from a .txt file you had in your home partition!).

    It takes away all the nonsense of moving data across drives upon a reinstall, and it it also eliminates the nonsense of having to import/export application data (such as bookmarks, emails, tasklists) and reconfiguring preferences “just the way things were” for a particular user.

    Particularly useful also for the Fedora computers I maintain for friends and family, where they sometimes have to jump multiple major OS versions, in which case I may just clean-install on / instead of going through multiple upgrades.

    Not to mention that it has allowed me to essentially keep the same disks since 2012 (a 2TB hard disk for /home, and a SSD for “/” (with a couple of symlinks into my home for the few folders where I need extremely low latency). At some point I replaced the SSD by a bigger one, and recently I clean-installed because I decided to remove all the dual-boot crap from it… and this was only trivial and “zero risk” to do because /home is on a separate drive.

    As for backups: I wrote rsync commands into a .sh that I launch as my backup script. I wrote those commands a bit over 15 years ago, and they still work. It usually backs up my 2TB /home (no big surprise here) to a USB drive in 1 to 5 minutes flat.

    1. Nobody’s saying that things which don’t scale can’t be useful if you invest the time (if they weren’t, why would people go through the hassle).

      But if the problem a separate home partition is solving is “I want to be able to replace my OS without re-configuring stuff”, we need a solution that doesn’t require people to know or care what partitions are, how to create them during install, set mount points, etc.

      You could probably design a somewhat scalable experience around separate home partitions by automating their creation and handling all the edge cases gracefully. Even in the best of cases you’d still end up with situations where the system sometimes says there’s not enough space to e.g. install apps, even though there seems to be plenty of space when they check in the file manager. It’s always going to be messy.

      My feeling is that if we had a proper OS backup/restore system which just stores data on an external drive that would be a cleaner solution, especially because it could handle re-installing apps and the like as well.

  6. I have mixed feelings about this. On one hand, I absolutely think that computers need to be accessible to the masses. On the other hand, that kind of individualized tinkering is how I got to the point where I’m able to contribute upstream in the first place.

    Certainly things should be polished out of the box, but I also think we need to focus on lowering the barrier to entry for modifying software. The more a user experiences the mutability of computing for themselves, the more they’ll begin to understand that software isn’t just something handed down to them from on-high, but rather something they can be an active participant in shaping.

    Emacs, once you’re over the initial learning curve, is a good demonstration of this philosophy. To use an example from my own experiences: I was using a music player in Emacs, and if I clicked somewhere in the blank lines after the last song in my playlist, it would switch to playing the last song. This was counterintuitive to me, so I went digging to try and change it.

    Emacs has very good inspection tools, so I was able to figure out what the mouse click was doing with “C-h k”, and – crucially – try out my changes to the running program on the fly. I didn’t have to deal with build systems, I didn’t have to track down the source code and rebuild to even start to look for a fix; I just edited the code running my system in-place. And then I contributed it upstream: https://github.com/pft/mingus/pull/27

    I’m a sysadmin but not a professional programmer, and the programs that I’ve contributed to the most are those that make their inner workings available like this. I’m certainly not saying that more programs should emulate Emacs’s UI; it’s a relic that predates modern UI conventions, after all. But more programs should be as introspectable and easy to change. That, I think, is how we can give people real control over their computers.

  7. I would oppose to any closed source and services that are known to deprive user’s life e.g. Google services, Windows. Also I would not recommend users visiting websites with all malicious scripts and advertising content on, trust other people’s computers (cloud services) for backup, do not use encrypted email because.. it does not scale?

    I cannot accept the fact that every millisecond of our life is getting systematically recorded and analysed to be later sold as batches of data to interest companies or other entities. Do you suggest that we should rely and use software and services from companies that are known to disrespect our life and diminish our freedoms for the sake of profit?

    Are you going to continue support these companies and their unethical revenue system just because smaller projects or software aren’t doing things that scale?

    We should not always aim to do things that scale but also things ensure our freedoms are not threatened, even if this means that is not so convenient.

    Many projects and software wouldn’t have existed if were only doing things that scale.

    1. Congratulations, you’ve missed the point entirely. Nobody’s saying people should use unethical software/services because the ethical alternatives we have currently don’t scale (I personally still do quite a few non-scalable things for this reason).

      What I’m trying to get across is that we should make alternatives that scale, because otherwise the only people who can benefit from them are a tiny minority of tech enthusiasts.

      In some cases that will mean burning down what we have and replacing it with something completely different (e.g. I’m quite sure encrypted email cannot be fixed), in others it’s probably enough to just set nicer defaults, integrate better, and focus on having kick-ass UX.

  8. I used to use fish as well but noticed it’s slower compared to bash (and zsh). So I switched to zsh, together with ohmyzsh. I haven’t tinkered much with ohmyzsh at all, just used randomly chosen themes for a short while and stopped when one looked goos enough. On the plugin side I haven’t touched it, the defaults are good.

  9. Hi, Tobias

    I liked this post a lot because it shows an approach, a way of living, beyond the individual hacker’s impulse. It gives a purpose, grows the commons and helps making more sustainable lives.

    I guess everyone has to live with an ever-evolving personal position on a gradient between the things you do for you (for need, for fun or just because) and the things you do with the expectation (or the negotiation) of being relevant to others.

    Glad to see you pointing towards the second, we need more people that dare to take on that challenge.

  10. This was a really interesting post. This was something I had been thinking about but couldn’t quite put into words yet. You want to optimize your workflow and make it as cool as possible, but at some point you burn out. I think there’s a sweet spot for how many things that don’t scale you can keep, but it’s different for everyone.
    I’m curious as to why auto-starting apps doesn’t scale tho

  11. This is also a strong argument for free software users paying for free software so that development resources can scale.

    We will never get the end user quality of experience that is provided by iOS, MacOS, Android, or Windows without those developer hours.

    Those developer hours need financial resources to produce enough output. The reason is that there is a large amount of mental overhead required before there is any productive output. A single developer working 40 hours a week will produce much more than 10 equivalent developers working 4 hours a week.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.