Effective mentoring programs

3:36 pm community, freesoftware, General, gimp, gnome

I’ve been thinking a lot recently about mentoring programs, what works, what doesn’t, and what the minimum amount of effort needed to bootstrap a program might be.

With the advent of Google Summer of Code and Google Code-In, more and more projects are formalising mentoring and thinking about what newcomers to the project might be able to do to learn the ropes and integrate themselves into the community. These programs led to other organised programs like GNOME’s Women Summer Outreach Program. Of course, these initiatives weren’t the first to encourage good mentoring, but they have helped make the idea of mentors taking new developers under their wing much more widespread.

In addition to these scheduled and time-constrained programs, many projects have more informal “always-on” mentoring programs – Drupal Dojo and GNOME Love come to mind. Others save smaller easier tasks for newcomers to cut their teeth on, like the “easy hacks” in LibreOffice. Esther Schindler wrote a fantastic article a few years ago documenting best and worst practices in mentoring among different projects.

Most mentoring programs I have seen and participated in don’t have a very good success rate, though. In this article, I look at why that is the case, and what can be put in place to increase the success rate when recruiting new developers.

Why most mentoring fails

Graham Percival, a GNU/LilyPond developer, decided in 2008 to run an experiment. At one point, Graham decided that he would quit the project, but felt guilty about doing so in one go. So he  started the “Great Documentation Project” to recruit a replacement documentation team to follow on after his departure. He then spent 12 months doing nothing but mentoring newcomers to get them involved in the project, and documented his results. Over the course of a year, he estimates that he spent around 800 hours mentoring newcomers to the project.

His conclusions? The net result for the project was somewhere between 600-900 hours of productivity, and at the end of the year, 0 new documentation team members. In other words, Graham would have been better off doing everything himself.

Graham found that “Only 1 in 4 developers was a net gain for the project” – that is, for every 4 apprentices that Graham spent time mentoring, only 1 hung around long enough for the project to recoup the time investment he put into mentoring. A further 1 in 4 were neither gain or loss – their contribution roughly equalled the mentor time that they took up. And the remainder were a net loss for the project, with much more time spent mentoring than the results could justify.

The GNOME Women’s Summer Outreach Program in 2006 had 6 participants. In 2009, the GNOME Journal ran a “Where are they now?” follow-up article. Of the 6 original participants, only one is still involved in free software, and none are involved in GNOME. Murray Stokely did a follow-up in 2008 to track the 57 alumni of Summer of Code who had worked on FreeBSD. Of these, 10 students went on to get full commit access, and a further 4 students were still contributing to FreeBSD or OpenBSD after the project. Obey Arthur Liu also did a review of Debian participants in 2008. Of 11 students from 2008 who had no previous Debian developer experience, he found that 4 remained active in the project one year later.

From my own experience as a replacement mentor and administrator in the Summer of Code for the GIMP in 2006, we had 6 projects, most of which were considered a success by the end of the summer, yet of the participating students, none have made any meaningful contribution to the GIMP since.

I feel safe in saying that the majority of mentoring projects fail – and Graham’s 1 in 4 sounds about right as an overall average success/failure rate. This begs the question: why?

Most mentored projects take too long

What might take a mentor a couple of hours working on his own could well take an apprentice several days or weeks. All of the experience that allows you to hit the ground running isn’t there. The most important part of the mentoring experience is getting the student to the point where he can start working on the problem. To help address this point, many projects now require Summer of Code applicants to compile the project and propose a trivial patch before they are accepted for the program, but understanding the architecture of a project and reading enough code to get a handle on coding conventions will take time. It will also take mentor time. It takes longer to teach a newcomer to your project than to do the work yourself, as anyone who has ever had a Summer intern will attest.

When you set a trainee task which you estimate to be about 4 hours work, that will end up costing a few weeks of volunteer effort for your apprentice, and 8 to 10 hours mentoring time for you during that time. Obviously, this is a big investment on both sides, and can lead to the apprentice giving up, or the mentor running out of patience. I remember in the first year of Summer of Code, projects were taking features off their wishlists that had not been touched for years, and expecting students new to the project to come in and work full time implementing them perfectly over the course of 12 weeks. The reality that first year was that most of the time was spent getting a working environment set up, and getting started on their task.

Mentoring demands a lot of mentors

As a free software developer, you might not have a lot of time to work on your project. Josh Berkus, quoted in Schindler’s article, says “being a good mentor requires a lot of time, frequently more time than it would take you to do the development yourself”.  According to the Google Summer of Code FAQ, “5 hours a week is a reasonable estimate” for the amount of time you would need to dedicate to mentoring. Federico Mena Quintero suggests that you will need to set aside “between 30 and 60 minutes a day“.

When you only have 10 hours a week to contribute to a project, giving up half of it to help someone else is a lot. It is easy to see how working on code can get a higher priority than checking in with your apprentice to make sure everything is on track.

Communication issues

More mentoring projects fail for lack of communication than for any other reason.

Apprentices may expect their mentors to check in on them, while mentors expect apprentices to ask questions if they have any. Perhaps newcomers to the project are not used to working on mailing lists, or are afraid of asking stupid questions, preferring to read lots of documentation or search Google for answers. In the absence of clear guidelines on when and how parties will talk to each other, communication will tend towards “not enough” rather than “too much”.

No follow through

Many mentoring programs stop when your first task is complete. The relationship between the mentor and the apprentice lasts until the end of the task, and then either the apprentice goes off and starts a new task, with a new mentor, or that is the end of their relationship with the project. I would be really interested to hear how many Summer of Code mentors maintained a relationship with their students after the end of the Summer, and helped them out with further projects. I suspect that many mentors invest a lot of time during the program, and then spend most of their time catching up with what they wanted to do.

Project culture

In her OSCON keynote in 2009, Skud talked about the creation of a welcoming and diverse community as a prerequisite for recruiting new developers. Sometimes, your project culture just doesn’t match newcomers to the project. If this happens regularly, then perhaps the project’s leaders need to work on changing the culture, but this is easier said than done. As Chris di Bona says in this video, “the brutality of open source is such that people will learn to work with others, or they will fail”. While many think that this kind of trial-by-fire is fine, the will not be the environment for everyone. It is really up to each project and its leaders to decide how “brutal” or forgiving they want to be. There is a trade-off: investing time in apprentices who will contribute little is a waste of time, but being too dismissive of a potential new developer could cost your project in the long run.

Mentoring best practices

Is all the effort worth it? If mentoring programs are so much hassle, why go to the bother?

Mentoring programs are needed to ensure that your project is long-term sustainable. As Graham says in his presentation: “Core developers do most of the work. Losing core developers is bad. Projects will lose core developers.” Do you need any other reason to start actively recruiting new blood?

There are a few simple things that you can put in place to give your mentoring program a better chance of success.

Small tasks

Mentored tasks should be small, bite-sized, and allow the apprentice to succeed or fail fast. This has a number of advantages: The apprentice who won’t stick around, or who will accomplish nothing, has not wasted a lot of your mentor’s time. The apprentice who will stay around gets a quick win, gets his name in the ChangeLog, and gains assurance in his ability to contribute. And the quick feedback loop is incredibly rewarding for the mentor, who sees his apprentice attack new tasks and increase his productivity in short order. Graham implies that a 10 minute task is the right size, with the expectation that the apprentice might take 1 hour to accomplish the task.

A ten minute task might even take longer to identify and list than it would to do. You can consider this cost the boot-strapping cost of the mentoring program. Some tasks that are well suited to this might include:

  • Write user documentation for 1 feature
  • Get the source code, compile it, remove a compiler warning, and submit a patch
  • Critique 1 unreviewed patch in Bugzilla
  • Fix a trivial bug (a one line/local change)

Of course, the types of tasks on your list will change from one project to the next.

Mentoring is management

Just as not everyone is suited to being a manager, not everyone is suited to being a mentor. The skills needed to be a good mentor are also those of a good manager – keeping track of someone else’s activity while making them feel responsible for what they’re working on, communicating well and frequently, and reading between the lines to diagnose problems before it’s too late to do anything about them.

When you think of it in this way, there is an opportunity for developers who would like to gain management experience to do so as a mentor in a free software project. Continually recruiting mentors is just as important as recruiting developers. Since mentoring takes a lot of time, it’s important that mentors get time off and new mentors are coming in in their place.

Pair apprentices with mentors, not tasks

An apprentice should have the same mentor from the day he enters the mentoring program until he no longer needs or wants the help. The relationship will ideally continue until the apprentice has himself become a mentor. Free software communities are built on relationships, and the key point to a mentoring program is to help the creation of a new relationship. Mentoring relationships can be limited in time also, 6 months or a year seem like good time limits. The time needed to mentor will, hopefully, go down over this period.

Regular meeting times

Mentors and apprentices should ensure that there is a time on their calendar for a “one on one” at regular times. How regularly will depend on the tasks, and the amount of time you can spend on it. Weekly, fortnightly or monthly are all reasonable in different situations. This meeting should be independent of any other communication you have with the person – it is too easy for the general business of a project to swallow up a newbie and prevent their voice from being heard. Rands said it well when he said “this chatter will bury the individual voice unless someone pays attention.”

Convert apprentices into mentors

Never do you understand the pain of the initial learning curve better than when you have just gone through it. The people best suited to helping out newcomers to the project are those who have just come through the mentoring program themselves.

This is a phenomenon that I have seen in the Summer of Code. Those students who succeed and stay with the project are often eager to become mentors the following year. And they will, in general, be among the best mentors in the project.

Keep track

For all involved, it’s useful to have some idea of the issues newcomers have – ensure that documenting solutions is part of what you ask. It’s also useful to know how successful your mentoring program is. Can you do better than the 1 in 4 success rate of LilyPond? Keeping track of successes and failures encourages new mentors, and gives you data to address any problems you run into.

Manage the mentors

All of this work has overhead. In a small project with 1 or 2 core developers, it’s easy enough to have each core developer take an apprentice under their wing, and co-ordinate on the mailing list. In bigger projects, keeping track of who is a mentor, and who is mentoring who, and inviting new mentors, and ensuring that no-one falls through the cracks when a mentor gets too busy, is a job of itself. If your mentoring program goes beyond more than ~5 mentors or so, you might want to consider nominating someone to lead the program (or see who steps up to do the job). This is the idea behind the Summer of Code administrator, and it’s a good one.

Go forth and multiply

Developer attrition is a problem in open source, and recruitment and training of new developers is the only solution. Any project which is not bringing new developers up to positions where they can take over maintainership is doomed to failure. A good mentoring program, however, with a retention rate around 25%, organised continuously, should ensure that your project continues to grow and attract new developers.

Replenishing your stock of mentor tasks and recruiting new mentors will take effort, and continual maintenance of someone putting in a few hours a week. If you execute well, then you will have helped contribute to the long term diversity and health of your project.

18 Responses

  1. Benjamin Otte Says:

    I’ve always wondered if mentoring is the wrong approach to get new contributors. Certainly mentoring was not what made me contribute.

    I also do remember a quote in some of the papers about GNOME where someone said, that important contributors don’t need (or want?) mentoring, they just show up and start contributing.

  2. Robert Ancell Says:

    Really nice write-up Dave!

  3. meme Says:

    I have been a GSoC student 3 times now (3rd is this year). Here’s my experience.

    I always hear some friends of mine that they are working with the mentor, or talking with them over the phone, IM, mail, yet this is so weird to me that I don’t even know where to start.

    Let’s take for example my last year involvment in GSoC. The actual mentor provided no feedback on my application. After being selected I sent the customary introduction email, got a response. I started sending weekly updates on the project, and various questions I had and got *NO* answer. At the end of the summer I got an email that the code looks clean and has been pushed to HEAD and it broke the build. Well d’oh, at least he could have asked me if it’s ok to do so.

    This year is the same.

    I know quite a few people in the same situation. So why would anyone stick when no one even bothers to read their e-mails? I remember sending a shitload of emails and man, the answers I got were like having 1000 monkeys at typewriters, or calling customer service.

    As far as I know most people mentoring in GSoC don’t have the time, or don’t care about the programs. The only people that do are the org administrators.

    Regards

  4. meme Says:

    The sad thing is that I have some (pretty much) experience with FLOSS software and online communities, but have never actually been really involved in one, so I’m used to this sort of crap in FLOSS communities (devs not bothering to read your bug reports, they do so only if they *know* you, they don’t bother to read, try, provide feedback to patches, rude attitude without providing any real feedback) but most people won’t take this crap that long just for the sake of free software.

    PS: I would like to see how other students perceive GSoC or other mentoring programs.

    Regards

  5. liberforce Says:

    Very nice article, Dave! Seeing how we could lower the barrier for newcomers (in general, not necessarily in mentoring programs) would also be interesting…

  6. ericb Says:

    Well, if you want to be credible, you should mention all the projects who did that …

    Why don’t you not mention the OpenOffice.org Education Project ?
    We mentor students since years, we wrote a lot of code and I even participated to both OOo and Go-OO with Google Summer of code.

    For the record :
    http://wiki.ooo4kids.org/index.php/WelcomeStudents

  7. ericb Says:

    Sorry, I forgot this important link, very :

    http://wiki.ooo4kids.org/index.php/ImproveMathEquationEditor

    (was : fix an 9 year OOo bug)

  8. Dave Neary Says:

    @Eric: Are you sure you meant to say “credible”? This sounds very harsh to me, as if everything I wrote is not credible, because I didn’t mention OOo.

    As I said in the article, lots of projects run ad hoc and organised mentoring. A partial list is here: http://teachingopensource.org/index.php/FOSS_Mentor_Projects – it would be impossible to list them all. I’m sorry I offended you by failing to mention OpenOffice.org education project.

    Dave.

  9. Graham Percival Says:

    @Benjamin, comment 1, re “import contributors don’t need
    mentoring, they just show up”.

    Most of the most prolific contributors to LilyPond did indeed
    simply “show up” and required no mentoring, but I estimate that
    about 20-30% of our “developer time” in 2011 came from people who
    would not have contributed without mentoring. Assuming that my
    guessimate is vaguely accurate, the question is: how much time did
    we lose on mentoring vs. how valuable is that 20-30% of “extra”
    developer time?

    There’s far too many variables (and estimates) floating around to
    make any kind of accurate judgement, but it’s certainly worth
    keeping that thought in the back of our minds. I *don’t* think
    that the amount of mentoring I did in LilyPond’s first big
    mentoring push was worth it[1], but I *do* think that LilyPond’s
    current policy/effort towards mentorship is less than it should
    be.

    [1] or rather: I think that the amount of mentoring was worth it
    as an experiment. It was not an effective way to spend developer
    time from the standpoint of getting development done, but I don’t
    regret performing the experiment.

  10. Jan Nieuwenhuizen Says:

    An interesting (or annoying, depending on your pov) detail of Graham’s mentoring effort was that iirc, none of his students was a real help with his Grand Documentation Project.

    Sadly (or luckily, depending on your pov), the ones who stayed became active as developers, rather than documentation writers.

  11. Elin Waring Says:

    I think FOSS people tend to be a bit over optimistic on this kind of outcome from mentoring programs. IF you are retaining 1 of 6 as committed contributors you are doing well. A lot of mentoring and internship programs are about discovering bad fit. For example, student teaching is important not just for learning how to teach but for students to realize that the classroom is not for them. Social work interns likewise really they do not have the stamina or patience for that work and the earlier the better. Similarly a lawyer friend from a non profit law firm told me that of each year’s class of summer interns they consider it great if they have one person who would really be “a keeper.” Sometimes it’s skill, sometimes it is poor match on organizational culture, sometimes students realize it is not for them. It’s far more powerful to think of mentoring programs as learning experiences for the students, whether or not they are technically students, and learning sometimes means things that are unintended or about themselves. Learning that you are a developer and not a documenter is a great thing to find out early rather than later.

    Just like you can’t expect every block of code you write to end up either working or getting committed to a distribution but you know that the time you spent on that code taught you something or helped you solve a problem, both mentoring and being a mentee have many possible outcomes all of which can be positive. (The exception being abuse on either side.)

    Sure, it is great to have best practices and they will help everyone have a better, less frustrating experience, they are absolutely helpful. But don’t judge their success based on he rate at which mentees are retained–it is not realistic or constructive to expect that number to be unrealistically high.

  12. Frederick Grose Says:

    It might help if mentors considered their role (from the dictionary) as “a wise and trusted counselor or teacher” for the mentee.

    Greedy or selfish mentors are not likely to “succeed”. But, if the mentor helps the mentee to discover their strengths and limitations, and a learning or career path, then success is quite likely (by that definition, at least), and the relationship is generally more rewarding for both parties. This maximizes the productivity of the the relationship.

    Success for the project might be maximized by careful and selective recruitment and placement, and rapid evaluation and redeployment as needed.

  13. Shaun McCance Says:

    Hi Dave. Thanks for the post. Well-researched and well-written. I actually disagree with your recommendation to do small tasks, at least for documentation tasks. While I’ve had mixed experiences with mentoring programs in general, I had nothing but bad experiences with Google Code-in. I won’t do that program again.

    The problem is that people think they can write good help just because they’re literate. Good technical writing takes some commitment to understand problems, research answers, and communite it all in a way people understand. The small tasks in GCI just encouraged people to slap some words on a page. Then I’m expected to give credit for content that I can’t even use.

    Bite-sized jobs are great when they actually get the ball rolling. I’ll usually hand out easy topics to new contributors. Then we do some back-and-forth to make their work better. They ramp up pretty quickly.

    But when you stick a dollar (pound/euro/whatever) amount on small documentation tasks, people seem to just crank out crap to get the money. They don’t seem interested in going through reviews and revisions. It never gets the ball rolling.

    I’m sure others have had successful experiences with GCI and similar bite-sized mentoring programs. But it my experience, it’s nothing but bad.

  14. Sumana Harihareswara Says:

    I’m a GSoC organizational mentor (for MediaWiki) and I am ensuring none of our students get the experience meme had, per comment #1.

    B. Otte: citation needed, and then once we see the actual quote, we can respond at greater length.

    I’d be interested to know how many mentees then shift their interests and go into different FLOSS projects.

  15. Eric Pritchett Says:

    Nice post! This reminds me of teaching dancing to someone who has never danced before. The goal of the first lesson is not so much teaching them the basics as it is to see them out on the dance floor after the lesson is done. This often has to just do with making them feel comfortable in their environment and more importantly make them want to come back. If they’re not on the dance floor chances are they’re not going to come back.
    Just like anything else there are bad teachers and good teachers. Reading the posts above it seems like there are a couple things missing on the mentor’s part and that’s responsibility and accountability. If you want anything to concede you need at least those two things.

  16. Graham Percival Says:

    Great to see more discussion about the difficulty of mentoring new
    talent! I’ve seen many programmer discussions about why they
    don’t contribute and people saying “projects should do more to
    encourage new developers”, but it’s not easy.

    (disclaimer: yes, I’m the Graham Percival of LilyPond which only
    managed the “1 in 4″ success rate)

    I’m still involved in LilyPond, although now I’m managing releases
    and doing general administration rather than documentation
    specifically. We have a Contributor’s Guide with specific
    enumerated reponsibilities for contributors (“newbies”) and
    mentors:

    http://lilypond.org/doc/v2.13/Documentation/contributor/mentors

    However, recruiting mentors is difficult (for all the reasons that
    Dave has already mentioned). I’m finding that new contributors
    are more succeptable to being guilt-tripped into being mentors –
    again, for the reasons that Dave has already mentioned.

  17. Neary Consulting » Effective mentoring programs Says:

    [...] Reposted from gnome.org [...]

  18. [M]etabrain [E]ntry [L]og » Blog Archive » Yup, I’m still alive. Says:

    [...] a sort of BarCamp for government instead of tech and new ventures. Dave Neary’s post on effective mentoring programs in FOSS and the study on sustainable development (in terms of contributor burnout and turnover, not [...]