Limbo: Why users are more error-prone with git than other VCSes

Limbo is a term I use but VCS authors don’t. However, that’s because they tend to ignore a certain state that exists in all major VCSes (and give it no name because they tend to ignore it) despite the fact that this state seems to be the largest source of errors. I call this state limbo.

How to make git behave like other VCSes

Most potential git users probably don’t want to read this whole page, and would like their knowledge from usage of other VCSes to apply without learning how the index and limbo are different in git than their previous VCS (despite the really cool extra functionality it brings). This can be done by

  • Always using git diff HEAD instead of git diff
  • and

  • Always using git commit -a instead of git commit

Either make sure you always remember those extra arguments, or come back and read this page when you get a nasty surprise.

The concept of Limbo

VCS users are accustomed to thinking of using their VCS in terms of two states — a working copy where local changes are made, and the repository where the changes are saved. However, the working copy is split into three sets (see also VCS concepts):

  • (explicitly) ignored — files inside your working copy that you explicitly told the VCS system to not track
  • index — the content in your working copy that you asked the VCS to track; this is the portion of your working copy that will be saved when you commit (in CVS, this is done using the CVS/Entries files)
  • limbo — not explicitly ignored, and not explicitly added. This is stuff in your working copy that won’t be checked in when you commit but you haven’t told the VCS to ignore, which typically includes newly created files.

The first state is identical across all major VCSes. The second two states are identical across cvs, svn, bzr, hg, and likely others. But git splits the index and limbo differently.

One could imagine a VCS which just automatically saves all changes that aren’t in an explicitly ignored file (including newly created files) whenever a developer commits, i.e. a VCS where there is no limbo state. None of the major VCSes do this, however. There are various rationales for the existence of limbo: maybe developers are too lazy to add new files to the ignored list, perhaps they are unaware of some autogenerated files, or perhaps the VCS only has one ignore list and developers want to share it but not include their own temporary files in such a shared list. Whatever the reason, limbo is there in all major VCSes.

Changes in limbo are a large source of user error

The problem with limbo is that changes in this state are, in my experience, the cause of the most errors with users. If you create a new file and forget to explicitly add it, then it won’t be included in your commit (happens with all the major VCSes). Naturally, even those familiar with their VCS forget to do that from time to time. This always seems to happen when other changes were committed that depend on the new files, and it always happens just before the relevant developers go on vacation…leaving things in a broken state for me to deal with. (And sure, I return the favor on occasion when I simply forget to add new files.)

A powerful feature of git

Unlike other VCSes, git only commits what you explicitly tell it to. This means that without taking additional steps, the command “git commit” will commit nothing (in this particular case it typically complains that there’s nothing to commit and aborts). git also gives you a lot of fine-grained control over what to commit, more than most other VCSes. In particular, you can mark all the changes of a given file for subsequent committing, but unlike other VCSes this only means that you are marking the current contents of that file for commit; any further changes to the same file will not be included in subsequent commits unless separately added. Additionally, recent versions of git allow the developer to mark subsets of changes in an existing file for commit (pulling a handy feature from darcs). The power of this fine-grained choose-what-to-commit functionality is made possible due to the fact that git enables you to generate three different kinds of diffs: (1) just the changes marked for commit (git diff –cached), (2) just the changes you’ve made to files beyond what has been marked for commit (git diff), or (3) all the changes since the last commit (git diff HEAD).

This fine-grained control can come in handy in a variety of special cases:

  • When doing conflict resolution from large merges (or even just reviewing a largish patch from a new contributor), hunks of changes can be categorized into known-to-be-good and still-needs-review subsets.
  • It makes it easier to keep “dirty” changes in your working copy for a long time without committing them.
  • When adding a feature or refactoring (or otherwise making changes to several different sections of the code), you can mark some changes as known-to-be-good and then continue making further changes or even adding temporary debugging snippets.

These are features that would have helped me considerably in some GNOME development tasks I’ve done in the past.

How git is more problematic

This decision to only commit changes that are explicitly added, and doing so at content boundaries rather than file boundaries, amounts to a shift in the boundary between the index and limbo. With limbo being much larger in git, there is also more room for user error. In particular, while this allows for a powerful feature in git noted above, it also comes with some nasty gotchas in common use cases as can be seen in the following scenarios:

  • Only new files included in the commit
    1. Edit bar
    2. Create foo
    3. Run git add foo
    4. Run git commit

    In this set of steps, users of other VCSes will be surprised that after step 4 the changes to bar were not included in the commit. git only commits changes when explicitly asked. (This can be avoided by either running git add bar before committing, or running git commit -a. The -a flag to commit means “Act like other VCSes — commit all changes in any files included in the previous commit”.)

  • Missing changes in the commit
    1. Create/edit the file foo
    2. Run git add foo
    3. Edit foo some more
    4. Run git commit

    In this set of steps, users of other VCSes will be surprised that after step 4 the version of foo that was commited was the version that existed at the time step 2 was run; not the version that existed when step 4 was run. That’s because step 2 is translated to mean mark the changes currently in the file foo for commit. (This can be avoided by running git add foo again before committing, or running git commit -a for step 4.)

  • Missing edits in the generated patch
    1. Edit the tracked file foo
    2. Run git add foo
    3. Edit foo some more
    4. Run git diff

    In this set of steps, users of other VCSes will be surprised that at step 4 they only get a list of changes to foo made in step 3. To get a list of changes to foo made since the last commit, run git diff HEAD instead.

  • Missing file in the generated patch
    1. Create a new file called foo
    2. Run git add foo
    3. Run git diff

    In this set of steps, users of other VCSes will be surprised that at step 3 the file foo is not included in the diff (unless changes have been made to foo since step 2, but then only those additional changes will be shown). To get foo included in the diff, run git diff HEAD instead.

These gotchas are there in addition to the standard gotcha exhibited in all the major VCSes:

How all the major VCSes are problematic

  • Missing file in the commit
    1. Edit bar
    2. Create a new file called foo
    3. Run vcs commit (where vcs is cvs, svn, hg, bzr…see below about git)

    In this set of steps, the edits in step 1 will be included in the commit, but the file foo will not be. The user must first run vcs add foo (again, replacing vcs with the relevant VCS being used) before committing in order to get foo included in the commit.

    It turns out that git actually can help the user in this case due to its default to only commit what it is explicitly told to commit; meaning that in this case it won’t commit anything and tell the user that it wasn’t told to commit anything. However, since nearly every tutorial on git[*] says to use git commit -a, users include that flag most the time (60% of the time? 98%?). Due to that training, they’ll still get this nasty bug. However, they’re going to forget or neglect this flag sometimes, so they also get the new gotchas above.

[*] Recent versions of the official git tutorial being the only exception I’ve run across. It’s fairly thorough (make sure to also read part two), though it isn’t quite as explicit about the potential gotchas in certain situations.

How bzr, hg, and git mitigate these gotchas (and cvs and svn don’t)

These gotchas can be avoided by always running vcs status (again, replace vcs with the relevant VCS being used) and looking closely at the states the VCS lists files in. It turns out bzr, hg, and git are smart here and try to help the user avoid problems by showing the output of the status command when running a plain vcs commit (at the end of the commit message they are given to edit). This helps, but isn’t foolproof; I’ve somehow glossed over this extra bit of info in the past and still been bit. Also, I’ll often either use the -m flag to specify the commit message on the command line (for tiny personal projects) or a flag to specify taking the commit message from a file (i.e. using -F in most vcses, -l in hg).

11 thoughts on “Limbo: Why users are more error-prone with git than other VCSes”

  1. Nice article, I gained a level in git (happy as hell with git since 1.5 btw) reading it 🙂

    I’d love to see you expand on –

    “””Additionally, recent versions of git allow the developer to mark subsets of changes in an existing file for commit (pulling a handy feature from darcs).”””

    in a subsequent article.

  2. When using git, I learn to pay close attention to the summary that’s included when git fires up the editor and prompt for the commit message. It quite nicely lists the files that are tracked but have unmarked changes.

  3. I’ll second Michel’s comment: if you actually read the status message “git commit” gives you, you can never have this problem. It will always tell you about what you included in the commit and what you did not include in the commit.

  4. Um, two people who didn’t bother reading the whole article but commented anyway? Maybe with a couple ads I could make my blog just like slashdot. 😉

    I already mentioned the status message given by ‘git commit’. Two problems: (1) The status message doesn’t exist for people who use the -F option (or the -m option), and (2) I think lots of users will tend to miss that message. Even after writing this post a month or so ago and being fully aware of the gotchas, I still accidentally hit some of these gotchas a few times since then even when I was using a plain ‘git commit’ despite the status message that was right there. Granted git reset or the –amend option to git commit make it really easy to fix, but these gotchas do increase the learning curve and it’s something people need to be aware of while working.

  5. Ugh, I hate how wordpress marks me as ‘newren’ instead of as ‘Elijah’ in my comments on my blog. Let’s see if the configuration option I tried to twiddle has any useful effect…

  6. Woohoo! I beat the nasty wordpress beast. [Nothing to see here in this or the previous comment, move along and read the others.]

  7. I disagree that git is more error prone… With svn, I always found myself accidentally committing stuff I didn’t intend to. Then, once I do that, svn makes it so damned hard to back out a mistake. With git, I specify exactly what gets committed. Even if I blow it, with git I end up committing too little… That’s far preferable to committing too much. And, with Git, if I do completely blow it, it’s trivial to fix the last commit.

    So, my experience is the polar opposite of yours.

  8. You should also add to the end a note that even though it’s easy to leave a file /edit from the commit, git makes it very easy to correct that:

    git add foo.c
    git commit # Whoops, forgot bar.c
    git add bar.c
    git commit –amend # re-make the commit with the correct content

    Botching a commit is so common (in every VCS) that the ability to re-visit and edit the commit or even the whole history is what really sold git to me.

  9. With bzr, you can use the –strict option with “bzr commit”, and it will refuse to commit if there are “unknown” files (i.e. files that bzr hasn’t been told to ignore or to track) in the tree.

    You can even make this your default behaviour by adding this to your bazaar.conf:

    [ALIASES]
    commit=commit –strict

    (Aliases are documented at http://doc.bazaar-vcs.org/latest/en/user-guide/index.html#using-aliases)

    If you do botch a commit, it’s easy to “bzr uncommit” and then redo the commit the way you intended.

  10. I have found similar things using git, but I have found myself getting used to it. Infact I think that having to explicity add all changes to the index makes you less likely to forget to add a new file, as you are used to doing a git add for changes. But I agree that it’s easy to forget and not commit enough stuff. The one thing that I absolutely love about git is the ability to go back through your commits and make changes. I have caused my repository to be uncloneable because I accidently committed a huge log file, and git-pack couldn’t pack it in usable RAM. All I had to do was filter back through the old commits and remove the file from the index, problem fixed. If I had done this with svn, I would be up the creek without a paddle in a major way. I have had other reasons to filter old commits to – I added a bunch of data files that I later found I never really needed. Put 10000 files in my repo without needing them – D’oh, but git lets me get rid of them as if they were never there. I know this functionality is dangerous, but it’s powerful, and I’m really glad of it. It feels like git was built by developers for developers and has many features that were built because someone said – oh I’d really like it if git go do this, and then someone made it do that. The other really awesome thing is git-svn. Wow – I can manage an svn copy with git – totally rad.

Comments are closed.