Gerrit Patch Review From The Command Line

September 25th, 2011 by markmc

OpenStack Nova switched from bzr and launchpad to github and gerrit on Friday. While I’m delighted the project is using git now, I’ve always found the gerrit UI to be a bit of a pain.

On IRC, Monty Taylor mentioned the gerrit command line interface which looked fairly interesting. Sure enough, you can actually review and approve a patch using this without ever touching the web UI. Below is an example of reviewing a Glance patch, but the same thing would work for Nova.

First, you obviously need to clone the repo:

$> git clone git://github.com/openstack/glance.git
$> cd glance

To make life a little easier, you can add a host alias to your SSH config:

$> cat >> ~/.ssh/config <<EOF
Host review
  Hostname review.openstack.org
  Port 29418
  User markmc
EOF

Then add the gerrit server as a git remote:

$> git remote add -f gerrit ssh://markmc@review/openstack/glance.git

Okay, now browse the patches needing review:

$> ssh review gerrit query status:open project:openstack/glance | less

Once you’ve picked a patch, take it’s Change-Id: and look at its patch sets and reviews:

$> ssh review gerrit query status:open project:openstack/glance \
         change:I27bb6b3951422ad32e5e0225765b1056c5b3ffc5 \
         --current-patch-set --all-approvals | less

Then, using the ‘ref’ in the output, you can fetch the patches into your repo and review them:

$> git fetch gerrit refs/changes/36/636/2
$> git checkout -b git-authors FETCH_HEAD

Once you’re ready to submit your review, you can do:

$> git checkout master
$> git branch -D git-authors
$> ssh review gerrit review --code-review +1 -m "'Looks good to me.'" cd9b3a0f2fb91d0d01606ef4bbd90cf8f29267da

That’s all pretty neat, but I’m missing how to go about doing a detailed review with comments inline with quoted sections of code. Perhaps if ‘gerrit review’ could take the review comments over stdin?

Git Rebasing (cont.)

February 26th, 2011 by markmc

As I said already, git’s interactive rebase tool is seriously useful for preparing a nice, cleanly split up series of patches. And, despite some people’s dire warnings, there’s no reason not to share an in-progress patch series using git so long as you take care to warn others against relying on your tree not rebasing.

Why would a patch series not be complete? One reason might be that a patch introduces a regression. As they say, you often have to break some eggs to make an omelette but, if you value the power of git’s bisection tool, you’ll want each individual patch to be regression free.

Okay, say you’re porting an application from one database framework to another. You might do a bunch of hacking to demonstrate the concept and then send that work out for comment. Only at this point will you go about figuring out polishing the work off and, finally, cleaning the changes up into a nice patch series.

This approach implies that the work will only stop rebasing quite late in the day. Which leaves a problem – how can you possibly collaborate with others if your tree is rebasing? How can you take patches to fix regressions? How can others help you clean up the series?

Here’s one suggestion, based on an approach Stephen Tweedie came up with when we were working together on a series of patches:

  1. Say your branch is called fluffy-piglet. You’ve pushed it out and asked for comments. Don’t rebase this branch again.
  2. Create another branch called fluffy-piglet-rebasing, basing it initially on fluffy-piglet.
  3. Tag both branches with e.g. a -v1 suffix, check that the trees in both tags are identical:
    $> git diff fluffy-piglet fluffy-piglet-rebasing
    $> git show -s --format='%t' fluffy-piglet fluffy-piglet-rebasing
    
  4. Push the rebasing branch and the v1 tags to your repo.
  5. If you wish to rebase and do some cleanup work on the patches, do so and tag and push the result to the fluffy-piglet-rebasing branch in your repo as in (3) and (4), but using a new suffix.
  6. If you receive some patches, pull them into your fluffy-piglet branch, tag the result and rebase the patches onto the rebasing branch e.g.
    $> git tag fluffy-piglet-v3
    $> git rebase --onto fluffy-piglet-rebasing-v2 fluffy-piglet-v2
    $> git tag fluffy-piglet-rebasing-v3
    
  7. If you wish to rebase unto latest upstream, you could first enable git’s “reuse recorded resolution” feature:
    $> git config --global rerere.enabled true
    

    Then you rebase the rebasing branch:

    $> git checkout fluffy-piglet-rebasing
    $> git rebase upstream/master
    $> git tag fluffy-piglet-rebasing-v4
    

    And then, you merge upstream into the non-rebasing branch:

    $> git checkout fluffy-piglet
    $> git merge upstream/master
    $> git tag fluffy-piglet-v4
    

    As in (3), you should be able to verify that the two resulting trees are identical.

    If any conflicts needed to be resolved during rebasing, there’s a good chance that having rerere enabled will mean the conflict will be automatically resolved when merging.

  8. Finally, if anyone wants to help you with any of the series cleanup work, just ‘pass the baton’. You basically say, ‘No more rebasing from me after v4, go ahed’ and the other person can work away on the rebasing branch until they are ready to pass control back again.

This certainly isn’t a straightforward workflow, but it gives you:

  • The ability to work with others since folks have a non-rebasing branch to work against
  • The ability to clean up a series using rebase while still having confidence that nothing is being screwed up because you have the pair of tags with identical tree contents
  • The ability to allow others to help clean up the series too

The fact that this workflow is so awkward has its advantages too – it encourages you to clean up the series early and stop rebasing it. This is not a workflow you’d like to use for an extended period of time.

Git Rebasing

February 23rd, 2011 by markmc

For me, ‘git rebase -i’ is perhaps git’s killer feature. I’m a big fan of small, self-contained commits both for ease of patch review and for the sake of useful commit history later. I used to do this on CVS using quilt but git takes a huge amount of the pain out of it.

Ever since I discovered the feature a few years ago, I’ve also been vaguely aware of kernel developers advice to people on rebase … often simplified to OMG no. Never rebase..

When pushed to elaborate, I guess most would say:

Once you share a commit with someone, never rebase it. They may base their work on your commit and by rebasing it, you’re screwing everything up.

One memorable comment from Linus on the subject was “Have the f*cking back-bone to be able to stand behind what you did!”.

In context, this all makes sense. If a kernel developer sends a pull request and it gets merged into one tree, then rebases and that gets merged into another tree and both get merged into Linus’s tree … then yes, you have a bit of a disaster on your hands.

However, I think the rules above are too simplistic for most git newbies. Such newbies are unlikely to see their trees pulled into the whirling vortex of kernel trees so there’s no need to terrify them about using rebase.

My advice is:

  1. If you’re learning git, take the time to understand the rebase command and, especially, the interactive option.
  2. If you’re working on a series of patches, it’s perfectly fine for you to share that series with others even if it’s not finished. That means later rebasing a commit you’ve shared with others.
  3. If you’re worried people might base their work on a commit you plan to rebase later, then you warn people by putting e.g. “v1″, “rebases” or “rebasing” in the repository or branch name.
  4. If someone does base their work on a commit you have rebased, then point them at the Recovering From Upstream Rebase part of git-rebase(1). It’s really not the end of the world, especially in the simpler cases.

Our Bizarre Posting Conventions

February 17th, 2011 by markmc

I’ve been around open source mailing lists for so long now that I tend to forget that our conventions around sending plain text emails, quoting replies, threading, etc. aren’t necessarily obvious to everyone out there.

That’s especially true for some of the folks working on RHEV-M, many of whom have come from a Windows development background. I eventually realized that saying “just do what everyone else does” wasn’t really good enough and spent some time documenting what all those little conventions are.

I’ve posted some guidelines to the rhevm-api wiki. What am I missing?

Error Handling

February 15th, 2011 by markmc

I came across a fairly typical error handling mistake today and thought I’d share it.

The code was something like this:

class Factory {
    static Foo getFoo() {
        try {
            // load a .properties file, read a key from it
            ...
            return new Foo(propValue);
        } catch (Exception ex) {
            log.error("Failed to find key in properties file: " + ex.toString());
            return null;
        }
    }
}

and the caller did e.g.

    Factory.getFoo().doSomething();

The thing to note here is that the caller is assuming that getFoo() method never returns null.

That’s probably a good assumption on the part of the caller. The factory method should always succeed unless there is a programmer error (e.g. the key does not exist in the .properties file) or a system error (e.g. an I/O error accessing the file). Neither of which the caller should be expected to handle gracefully. That’s what exceptions are for!

In the event of an error, we’d get a NullPointerException from the caller site and an error printed to the logfile. If, for example, you are running this code in a debugger, you’ll be looking at a NullPointerException with no clue as to what caused the exception.

The moral of the story here is to think clearly about when errors should be handled gracefully and when you should just let exceptions be passed back up the stack.

In the end, I added a new exception type:

class FactoryException extends RuntimeException {
    
    private static final String ERROR_MSG = "Failed to lookup key {0} in file {1}";

    private String propsFile;
    private String propsKey;

    FactoryException(String propsFile, String key, Throwable cause) {
        super(MessageFormat.format(ERROR_MSG, propsFile, key), cause);
        this.propsFile = propsFile;
        this.propsKey = propsKey;
    }

    String getPropsFile() {
        return propsFile;
    }

    String getPropsKey() {
        return propsKey;
    }
}

and then re-factored the original code:

class Factory {
    static Foo getFoo() {
        try {
            // load a .properties file, read a key from it
            ...
            return new Foo(propValue);
        } catch (Exception ex) {
            throw new FactoryException(propsFile, propsKey, ex);
        }
    }
}

so now if there is an error, rather than getting a fairly useless NullPointerException you get:

FactoryException: Failed to lookup key foo in file bar.properties
    ...
Caused by: java.io.FileNotFoundException: ...

If Knowledge is Power …

January 16th, 2011 by markmc

If knowledge is power, then sharing your knowledge empowers others.

"Illumination"

Illumination by Mr. Skinner

In the open-source world, each of us endeavours to strengthen the communities to which we contribute. There is no better way to strengthen a community than empowering your fellow contributors with your knowledge.

There are no excuses in the open-source world for not sharing your knowledge. Our tools and processes are designed to make this happen naturally.

Having discussions on a mailing list allows other contributors to benefit from the information being shared in the discussion. It also ensures that information is archived for future contributors.

Detailed commit messages ensures that current and future developers can fully understand your reasoning for a given change. Pushing your in-progress work to a public branch allows others to take the work on and finish it, even if you don’t have time.

Thinking out loud in bugzilla means that even if your work is not complete on the problem, others can run with your initial ideas or analysis.

Using a public wiki for note taking and giving others access to your jumbled up thoughts, ideas, hints and tips is a great way to open your brain to the world.

A nice side-effect of all this is that if your knowledge is shared openly and archived, you don’t need to have such a good memory! :-)

RISE UP & SHARE

RISE UP & SHARE by alyceobvious

In other contexts, it might be good practice to jealously guard your knowledge. If you’re the guy that everyone has to come to because you’re the one guy who knows how to do something, that’s good for your job security, right? Surely, if you share your knowledge freely, that means anyone can do your job?

This is the exact opposite to what should motivate you as you contribute to a community. You absolutely do want to make it possible for others to come along and kick your ass. If somebody appears out of nowhere and starts doing your job better than you could ever have done, then that is an absolutely awesome outcome!

REST API For RHEV-M

June 21st, 2010 by markmc

Today, Eoghan Glynn and I announced the first milestone release of a REST API for Red Hat Enterprise Virtualization Manager.

The only current API for RHEV-M is a Windows Powershell plugin which provides a perfectly fine scripting interface for RHEV-M on Windows, but isn’t so easy to call remotely or to integrate with another application. By adding a REST API, we’re adding an integration interface which we hope everyone will find convenient to use.

If you have a 2.2 installation of RHEV-M, it’s a quick and painless process to download our distribution, deploy the WAR to an Java EE application server (e.g. JBoss EAP or AS) and play around with our Apache Felix Karaf based shell. You can also read the API reference guide and jump on our mailing list to give us feedback.

RHEV-M isn’t yet an open-source project, so we’ve had to put a lot of effort into making it possible to develop this REST API in the open. We’ve concentrated our initial efforts on API design and building a prototype implementation of the API on top of the Powershell interface in 2.2. However, in time for the next release of RHEV-M, our plan is to add a compatible implementation of the API directly to the RHEV-M backend. At that time, the API will be a fully supported part of the product.

If you care about integrating with RHEV-M, now is the time to get involved with the API project. While the API is already quite well defined, there is buckets of scope for design changes and adding new features. And, most of all, we want your help!

Porting RHEV-M from C# to Java

April 12th, 2010 by markmc

Lately, I’ve been doing what I can to help finish the massive effort to port RHEV-M from C# to Java. Check out Livnat Peer’s blog describing the various approaches considered and how the team settled on using a semi-automatic conversion process.

Reporting Fedora Virtualization Bugs

January 14th, 2009 by markmc

When reporting bugs against Fedora, the BugsAndFeatureRequests wiki page is a great place to get some information on the kind of things you can do to provide useful information in the bug report.

I’ve just added a page on reporting virtualization bugs. Hopefully it’ll help people find narrow down bugs, log files, get debug spew etc.

And, of course, it’s a wiki page – so go ahead and add your own tips!

Checksums, Scatter-Gather I/O and Segmentation Offload

May 28th, 2008 by markmc

When dealing with virtualization and networking in the kernel, a number of fairly difficult concepts come up regularly. Last week, while tracking down some bugs with KVM and virtio in this area, decided to write up some notes on this stuff.

Thanks to Herbert Xu for checking over these.

Also, a good resource I came across while looking into this stuff is Dave Miller’s How SKBs Work. If you don’t know your headroom from your tailroom, that’s not a bad place to start.

Checksumming

TCP (and other protocols) have a checksum in its header which is a sum of the TCP header, payload and the “pseudo header” consisting of the source and destination addresses, the protocol number and the length of the TCP header and payload.

A TCP checksum is the inverted ones-complement sum of the pseudo header and TCP header and payload, with the checksum field set to zero during the computation.

Some hardware can do this checksum, so the networking stack will pass the packet to the driver without the checksum computed, and the hardware will insert the checksum before transmitting.

Now, with (para-)virtualization, we have a reliable transmission medium between a guest and its host and any other guests, so a PV network driver can claim to do hardware checksumming, but just pass the packet to the host without the checksum. If it ever gets forwarded through a physical device to a physical network, the checksum will be computed at that point.

What we actually do with virtualization is compute a partial checksum of everything except the TCP header and payload, invert that (getting the ones-complement sum again) and store that in the checksum field. We then instruct the other side that in order to compute the complete checksum, it merely needs to sum the contents of the TCP header and payload (without zeroing the checksum field) and invert the result.

This is accomplished in a generic way using the csum_start and csum_offset fields – csum_start denotes the point at which to start summing and csum_offset gives the location at which the result should be stored.

Scatter-Gather I/O

If you’ve ever used readv()/writev(), you know the basic idea here. Rather than passing around one large buffer with a bunch of data, you pass a number of smaller buffers which make up a larger logical buffer. For example, you might have an array of buffer descriptors like:

    struct iovec {

        size_t iov_len;     /* Number of bytes to transfer */

        void  *iov_base;    /* Starting address */

    };

In the case of network drivers, (non-linear) data can be scattered across page size fragments:

    struct skb_frag_struct {

        struct page *page;

        __u32 page_offset;

        __u32 size;

    };

sk_buff (well, skb_shared_info) is designed to be able to hold a 64k frame in page size[1] fragments (skb_shinfo::nr_frags and skb_shinfo::frags). The NETIF_F_SG feature flag lets the core networking stack know that the driver supports scatter-gather across this paged data.

Note, the skb_shared_info frag_list member is not used for maintaining paged data, but rather it is used for fragmentation purposes. The NETIF_F_FRAGLIST feature flag relates to this.

Another aspect of SG a flag, NETIF_F_HIGHDMA, which specifies whether the driver can handle fragment buffers that were allocated out of high memory.

You can see all these flags in action in dev_queue_xmit() where if any of these conditions are not met, skb_linearize() is called which coalesces any fragments into the skb buffer.

[1] – These are also known as non-linear skbs, or paged skbs. This what “pskb” stands for in some APIs.

Segmentation Offload

TCP Segmentation Offload (TSO). UDP Fragmentation Offload (UFO). Generic Segmentation Offload (GSO). Yeah, that stuff.

TSO is the ability of some network devices to take a frame and break it down to smaller (i.e. MTU) sized frames before transmitting. This is done by breaking the TCP payload into segments and using the same IP header with each of the segments.

GSO is a generalisation of this in the kernel. The idea is that you delay segmenting a packet until the latest possible moment. In the case where a device doesn’t support TSO, this would be just before passing the skb to the driver. If the device does support TSO, the unsegmented skb would be passed to the driver.

See dev_hard_start_xmit() for where dev_gso_segment() is used to segment a frame before passing to the driver in the case where the device does not support GSO.

With paravirtualization, the guest driver has the ability to transfer much larger frames to the host, so the need for segmentation can be avoided completely. The reason this is so important is that GSO enables a much larger *effective* MTU between guests themselves and to their host. The ability to transmit such large frames significantly increases throughput.

An skb’s skb_shinfo contains information on how the frame should be segmented.

gso_size is the size of the segments which the payload should be broken down into. With TCP, this would usually be the Maximum Segment Size (MSS), which is the MTU minus the size of the TCP/IP header.

gso_segs is the number of segments that should result from segmentation. gso_type indicates e.g. whether the payload is UDP or TCP.

drivers/net/loopback.c:emulate_large_send_offload() provides a nice simple example of the actions a TSO device is expected to perform – i.e. breaking the TCP payload into segments and transmitting each of them as individual frames after updating the IP and TCP headers with the length of the packet, sequence number, flags etc.