Back when I was at the University of Hannover working on my PhD, I was also involved in teaching. One of the most memorable experiences was organizing and managing the “software project” course four years in a row.
Every year, multiple teams of five students each went through a waterfall process to create software for a customer. This course wasn’t about programming, really: it was about working on a team, communication, and good practices. That’s what made it so memorable.
Students encountered challenges new to them, such as passionate technical arguments, apathetic team mates, or customer opinions that after a week would change under their feet.
However, two years of lectures in computer science had prepared them well for the technical challenges — or so I thought …
This is Frustrating!
For me as the organizer of the course, one of the most frustrating challenges was the apparent ineffectiveness of our prior courses. In the semesters leading up to the software project, we had taught our students about talking to customers, prototyping, architecture, and so on. But many seemingly forgot most of that the second this new course started.
The most frustrating experience I had was with students’ use of version control. Since it’s such a central tool in collaborative software development, at least using some of the best practices can go a long way to help a team coordinate.
But every year, when we looked through the students’ repositories, we would find that some students hadn’t committed anything at all! Many commits didn’t have a commit message (we used Subversion). And even if there was one, more often than I would’ve liked it would just describe the commit as “assorted changes” or “a bunch of fixes”.
This made it hard to understand what students had done and why.
It was frustrating when we wanted to continue development on the student projects. But to me, the most painful realization was that what we had taught them about using version control:
One commit should contain some isolated thing, like a bug fix or a new feature. Describe what you did in your commit so those coming after you will understand what you did and why. Keep your commits reasonably small.
… hadn’t made it from theory into practice for many of the students.
I was not frustrated or angry with the students, mind you. I was frustrated because I realized that the lectures in which we taught version control practices seemingly weren’t effective in changing student behavior for the better.
Persuasive Patterns
Luckily, such frustrating situations were exactly the problem I was trying to solve in my dissertation. I had been collecting persuasive patterns to nudge people to adopt certain practices from software development, and I had also developed a process that prescribes how to choose and apply these patterns.
So I went to work. I chose patterns that were relevant to developers not committing enough and forgetting to put messages in their commits. Then I created a Web application called Teamfeed that implemented those patterns. The following fall semester, I made Teamfeed accounts for all the students from the software project course.
I’ll tell you what happened soon — but first, let me show you the five patterns I chose and what Teamfeed actually did.
1. Normative Behavior
This is what Teamfeed looked liked:
Teamfeed showed the students’ commits in a newsfeed. Just like on Twitter or Facebook — however, the posts weren’t editable but got their text from commit messages. Students were “friends” with all their team mates and weren’t able to see anyone not on their team. When a student committed something without a message, Teamfeed would show a post simply stating “no message” in red letters.
2. Competition
To go along with the commits, Teamfeed also showed a leaderboard. It ranked students by their number of commits. When the semester was over, we interviewed the student teams. A memorable quote was:
“You only fixed that bug to gain a commit!”
Ha! 🙂
I hadn’t looked at the raw data yet at this point, but comments like this one gave me a clue that something might have worked.
In addition, students told me that the leaderboard made it very easy to spot team members who hadn’t yet committed anything at all.
“You see who committed how much. It’s a heuristic for whether someone’s actually collaborating.”
Would we have any non-committers this year? I was getting more and more curious.
3. Setting Goals
This pattern has its roots in goal setting theory. When goals are challenging but attainable — and you also get feedback on your progress towards the goal — they can be very motivating.
When you reached a certain number of commits in Teamfeed — your first commit, five commits, ten, 25, 50, and so on — you’d get a notification via email. Those would congratulate you for your progress. I also showed such milestones in the newsfeed using a different background color. The number of commits required to reach the next milestone was growing somewhat exponentially so reaching a milestone was perceived as harder or more valuable than for the prior one.
In an interview, a student told me:
“Milestones were useless. They lead to unnecessary commits. I committed things I could’ve committed later.”
Which, of course, was the whole point. When working with others, it can help pushing the things you’ve finished, as the likelihood of double work and merge conflicts will go down.
I also introduced milestones for teams. Competition isn’t necessarily the right social mechanism for all situations, and through team milestones I wanted to add an element of cooperation to it.
At the end of the semester, one of the teams had reached the 1000 commit milestone. In their interview, they told us how happy they’d been about that.
4. Triggers
Just as described in BJ Fogg’s Behavior Model and Nir Eyal’s Hooked [commission earned], I had recognized that triggers are an essential element for behavior change. So every Sunday afternoon, every team member would get an email that summarized the team’s achievements of the week. Combined with the milestone emails, these reminded students of Teamfeed’s core feature: counting and ranking their commits.
5. Progress
At the same time, these emails made a team’s progress visible. One student told us:
“It motivated you because you see things moving forward. You see progress.”
There has been ample research that has shown progress to be one of the most valuable motivations at work. In their book “The Progress Principle” [commission earned], Teresa Amabile and Steven Kramer summarize the findings of their studies showing exactly that.
Behold The Statistical Significance!
When the semester was over, I compared the students’ behavior with those of previous years. My favorite result: as opposed to prior semesters, there were no students with no commits at all! But I was similarly happy about these other observations as well, all of which were statistically significant (p < 0.05):
- the number of commits per student increased by 76%
- the number of commits that had messages per developer increased by 213%
- the ratio of commits with messages to overall commits increased by 75%
- the median time between commits increased by 44% (so commits were less of a binge and more of a regular activity)
- the length of commit messages increased by 28%
I’m not claiming that this experiment solves the problem completely or that significance levels prove anything. In experiments like these, there can always be confounding factors you don’t see. But all of the results above strongly indicate that something worked differently in this cohort, and — especially combined with the qualitative feedback I got in the interviews — it seems likely that Teamfeed influenced the students’ commit behavior.
We also looked at the commits and their messages of this cohort. After all, it was possible that some students were trying to cheat to gain more commits. So we manually went through their commits — the messages, the files they changed — and made sure those were all legitimate commits. And they were.
However, one student told us he felt like he was cheating. He couldn’t remember from the lecture anymore how large or small commits should be — which was exactly what triggered me to try this in the first place. So he made more commits than he would have normally because he was trying to reach the next milestone!
“When I was at 90 commits I made more and smaller commits, as I suspected there to be a milestone at 100. […] So I committed small fixes immediately instead of committing the fixes of a whole hour together.”
He felt like he was cheating because he couldn’t remember the best practice anymore.
So what’s the gist?
Was this manipulation? I don’t think so. I offered the students an additional tool to make version control more accessible. It lowered the barrier to looking at your project’s commit history. It helped lift the novices up a bit, while it left the more experienced developers alone — those actually told us that they set up email filters so they wouldn’t see the Teamfeed emails.
One of the goals of development processes is bringing everyone up to a certain baseline level. As an alternative, persuasive interventions such as Teamfeed do the same — without having to mandate anything, and without impacting the top performers (who can easily feel crippled by too much process).
Is it always best to commit as frequently as possible? Of course not. But for novices working on a team, I’d argue that committing too often is pretty much always better than committing too rarely. Sure, there might be commits that aren’t polished — but you’ll have fewer merge conflicts and coordination problems when everyone is regularly committing what they’ve been working on.
And it wasn’t all roses. One student told me:
“In this context, I’m critical about competition, because … it doesn’t say much.”
I agree, in part. For many situations, competition isn’t right. It can exclude others who prefer to cooperate. It can demotivate those at the bottom of the list. If I were to repeat such an experiment, I would probably choose a less competitive mechanism.
Note: If you want to read the academic accounts of this experiment, take a look at our paper “It Was a Bit of a Race: The Gamification of Version Control” and my PhD dissertation “Improving the Adoption of Software Engineering Practices Through Persuasive Interventions”; a catalog of persuasive patterns is in chapter 7, the experiment is in chapter 8.
Teamfeed helped me reinforce things we had already taught in lectures but couldn’t get students to adopt in practice. Personally, it made me very happy to see this discrepancy become smaller. And I hope it helped at least a few students get into the habit of committing earlier and more often.
But I think there is a greater lesson in this experiment. The way we design and choose our tools has a very direct impact on how we work. We should take some time to make sure we’re making the right choice.
Great tools make wanted behaviors obvious, easy, and desirable. Design them so; don’t leave it to chance. Tweet this!
Are you building a tool? Is there a behavior you want your users to adopt? Let us know in the comments and share what you’re doing.