On Testing Culture in GitHub Projects

Previous research suggests that the publicity on GitHub that is making developers’ actions and interactions more visible might have an effect on how software development practices are communicated and how they diffuse in projects.

My colleagues (Raphael Pham, Olga Liskin, Fernando Figueira Filho, Kurt Schneider) and I wondered: which influence does this have on testing practices? Does this create new challenges, and if so, how do developers cope with them? Which strategies do members of GitHub use to create a beneficial testing culture in their projects? To investigate this, we conducted interviews and questionnaires with diverse users of GitHub.

Update: our paper has been accepted for publication by ICSE 2013!

Results

We found several interesting phenomena that may support or inhibit testing practices in GitHub projects. For example:

1. The high social transparency of GitHub supports newcomers in learning the conventions of a project. They might not always read contribution guides; simply watching discussions on pull requests sometimes is enough. Just by watching how other, more senior project members behave, they learn what a good commit looks like.
2. Infrastructure with low barriers seems to be very important in getting developers to test their contributions. For example, Travis CI is a very simple way to add continuous integration to a project. Combined with existing tests that new contributors can simply copy and modify for their own submissions, the barrier to providing tests can be lowered significantly.
3. Because contribution has become so easy, project owners reported seeing what they called drive-by commits. Previous research reported that developers slowly progress from passive participation in the periphery of a project, over filing bugs, to fixing bugs, to at some point adding features as a core member. Drive-by commits, however, leave the contributor pretty much uninvolved with a project, let alone its culture.
4. Some more senior participants reported seeing a change in how tests are written. Some communities now take for granted that new developers learn how to write tests, and they make that very easy by providing clear guidelines and simple frameworks for testing. However, they heavily promote BDD and may put less of a focus on testing edge cases. Especially those learning testing in this environment write tests to make sure that a program behaves a certain way, and forget that they might also need to test how it should not behave (e.g. on invalid input).

In future work, we plan to dig deeper into some of the issues we found. For example: how do different developer personalities in a project influence the strategies that promote “good” testing behavior? What are the circumstances that make them successful? When do they do more harm than good?

Read our Report!

We published our research as a technical report (PDF): “Creating a Shared Understanding of Testing Culture on a Social Coding Site”.