git-mirror.gnome.org

As most of you know, I run bzr-mirror.gnome.org in my spare time. This was to allow people to try out the awesomeness of Bazaar as easily as possible. As much as I love Bazaar, certain outside forces mean that I sometimes have to use Git. So a few weeks ago when no one had approached bkor to set up git-mirror.gnome.org, I decided I would.

The plan:

  • Follow the example in the git-svn manual, 520 times
  • Run the equivalent of “bzr pull” everytime a commit happens (yay svn-commits-list)
  • Throw gitweb at it
  • Profit

All in all it went OK. Here are some things I encountered.

  1. I converted conduit, f-spot, tomboy and a few larger modules like gtk+. Then I tried to clone them over http. Found out I have to call git-update-server-info in post-update hook. This is well documented, and no big deal, but a surprise as its not needed with Bazaar.
  2. My initial attempt was cut short by James, who spotted I was missing an important argument (–rewrite-root) in my git-svn cloning. I had to abort the import after 3 days. Rip up and retry. Or so we thought. After some talking in #git, we found that –rewrite-root http://anything works, making me wonder why the root URL is stored in the log at all.
  3. When I ran the import again I found some of the modules randomly aborted mid conversion. It was able to resume them manually – yay, because it was one of the bigger modules the first time (gimp).
  4. I also found that gitweb wasn’t showing the tags and branches. James saved me again, this time with a script from Tim Janik at Imendio. Its full of perplexing git incantations, but works perfectly. It has to run after each change to keep everything up to date.
  5. I couldn’t get git-svn to work on a clone. My hero James was MIA, and I was struggling. A trip to #git offered mixed results. There were patches, there were more perplexing git incantations. Nothing seemed to work. With Robs help we got it working, and when James returned we got it working with branches and tags too. Day was saved. HeroPoints++.
  6. My version of git-svn chokes on accerciser and fast-user-switch-applet. My hero might yet fix this one too, as his git-svn was able to convert the problematic tags (this is due to cvs2svn).
  7. Now the mirror is ticking along, but gtk+ won’t update properly. I can update it from a local rsync, but not over http. I get “Apache got a malformed URI: REPORT request failed on ‘/svn/gtk+/!svn/vcc/default’: Unusable URI: it does not refer to this repository at /usr/bin/git-svn line 3833″.
  8. gnomemm Uses a non standard layout (at least in GNOME terms) which git-svn doesn’t like that much, so I will have to special case it in my scripts.
  9. I still have to go back and repack all the repositories as git-svn generates packs that are excessively large, and git gc –prune –agressive isn’t enough (thanks to Pieter for this advice).

After a week of churning its nice to have my server back to normal load. I still have to try and skin gitweb so that it looks GNOMEy and there are a hundred and one other things bothering me, but now you guys can:

cd ~
git clone http://git-mirror.gnome.org/git/cheese
cd cheese
git svn init -s svn+ssh://svn.gnome.org/svn/cheese --prefix=origin/svn/
git svn rebase

I’m not sure what will happen if you pull from the mirror and from svn.gnome.org. That is the ideal use case IMO (should be faster for you as mirror does the conversion for you). If someone wants to try and let me know how it handles it…

Want a mirror for your $dvcs? Ping me, maybe i’ll give it a go.

Tags: ,

9 Responses to “git-mirror.gnome.org”

  1. For anyone reading this as a HOWTO: a git mirror such as the one that has been set up is intended to help you speed up what would otherwise be a multi-hour long process of cloning the entire git history of a module. After cloning this mirror, your remote is still set to the SVN server and you still use git-svn. The mirror just makes it that much easier to get off and running with git-svn. It’s just as though you did git-svn init yourself.

  2. James Sharpe says:

    John: the gtk+ error is a bug in git-svn its apparently fixed in git HEAD.
    Jason: you are correct but you don’t have to use git-svn, you can continue to pull from origin and it’ll receive updates. You’ll need to use git-svn though if you want to commit back to the svn repo, that’s the whole point of having to init the git-svn tracking again.
    So in answer to Johns q regarding pulling from origin and svn-it just works, since the sha1 hashes should be identical from either source.

  3. halfline says:

    Awesome. Thanks for the effort and time.

  4. menko says:

    Thanks for the effort, sounds like much work.

    Not sure whether it is good to setup one mirror after another, why not just make the step and decide on one?

    Besides, tried http://hjemli.net/git/cgit/ for the web page? Actually it’s damn slow…

  5. John Carr says:

    @menko: Its not up to me to make up a decision and decide on one, all I can do is help make sure the decision is an informed one. I know some developers in GNOME like mercurial, but no one is providing them with any voice or support. But they contribute to GNOME too and deserve to have their opinion considered.

    I’ll have a look at cgit, as gitweb isn’t really up to the task of showing summaries for 520 modules yet. Its these kind of scaling problems that we can only see by using the tools, i guess. Hopefully cgit is packaged for etch-backports…

  6. This is all kinds of awesome. I set up the fredesktop cgit install so if you want help with that, track me down on IRC: krh on freenode.

  7. blah says:

    Any chance you can make this available via git:// instead of http://? It’s way faster.

  8. menko says:

    > I know some developers in GNOME like mercurial, but no one is providing them with any voice or support.

    That’s true. I personally also like mercurial much better, because of its much more consistent UI, but it has less supporters in the public…

    But if I am allowed, I will request such a mirror.

    Just to have another comparison on how it went, and how it does scale compared to git and bzr.

  9. Muelli says:

    Thanks for your efforts :)

    I’d love to see a mercurial mirror. I’m also willing to help setting it up :)