Practices

Continuous integration is a software system, like any other system is most useful when its users know how to use it. As CI systems are mostly used by a group of people, they tend to set some rules of the game. These rules are mostly informal and sometimes enforced by tooling around CI. These rules are formulated based on productivity and effectiveness of the whole team. Lets look at some such practices.

Committing no more than a fix on broken build

In any disciplined development team, this is the first practice which everyone needs to follow. When the integration build is broken there are only two things the commiter can do, commit a fix or revert her commit.

Why?

Continuous integration system start the build (almost) as soon as a check-in arrives. In other words it doesn't wait for too many check-ins to be batched up before it starts. This allows easier resolution of the question, "what broke the build?". This question gets harder to answer with increasing number of commits. Further commits (not fixes) on a broken build could potentially introduce even more failures, in turn making it tougher to fix the build without spending a lot of time and causing a lot of wait for everyone.

What if the integration is broken because of environmental reasons?

Continuous integration might also fail because the build agent is not configured right or there is dependency on network, external software etc. While in such situations there is hardly much the commiter can do than to force the build. At the same time, it is important to re-think the integration strategy/approach to reduce such failures.

What if the integration build succeeds randomly?

This is extremely common except that its not random. Software might seem to fail for random reason when the cause is unknown, but there is always a reason. In such cases it is recommended that the commiter seeks help (if required) to fix the randomness. For example this can be, because of new machine scenario is not handled in build scripts, timing issues in the tests being run, etc.

How do I decide between revert and fix?

Go back to basics. Which one would be productive for the team. There are three parameters: time to revert--commit--get-a-green-build, time to analyze-and-fix and special case with your commit. When not sure revert and commit-of-revert is a better idea.

Should I commit my other changes along with the fix?

Since committing code is a team activity (although performed by individual) hence your other changes are like commit performed by anyone else. So, the short answer is no.

Finally, the temptation to ignore small glitches, minor failures, is dangerous and puts you on slippery slope. Continuous integration status should be clear, and not left to interpretation of individuals.

Forcing a broken build without analysis

Continuous integration build runs when a commit is made. While this is indicates the status of latest change, it also means that running same build again and again without any change should get the same result. But as we have alluded to it earlier, this might not be the case always. The environment under which the integration is running also has an impact on the result apart from the code base. This means that the interplay between codebase and environment is also important to understand and to keep in order. Hence if the build fails because "there was a process holding lock to a file which it was trying to delete", needs to be fixed, or else it would fail again in future. Forcing a build might fix it but over time you would see that there are a lot of issues in the build caused by interplay between environment, codebase and the build application. Such builds are less reliable and hence don't get much attention to when they fail. There is a philosophy in lean, spend more time now to spend less later.

Good Practices

The practices listed below are simple to follow but they are really effective. [ref: Continuous integration states]

Claim responsibility of broken build

When you commit code, keep an eye on what happens to the continuous integration build [most continuous integration products come with some sort of push based notification tool, e.g. CCTray]. When the integration build fails, other members of team might get affected, hence the team should devise a mechanism for claiming responsibility of a failed build. This can be something like wearing a funny hat, indicating by volunteering to fix build in cctray or just shouting out loud claiming responsibility.

Leave the build green at the end of the day

In a way this is extension of, claiming the responsibility and fixing the build. Still it is important to stress because sometimes build take longer to fix and one might be tempted to come the next day and fix it. Most projects I have been on there are some people who commit at 2 am or at 7 am. So, if I come to office next day at 9am then it is not good enough for these people. As a thumb rule, do not make large commits (especially at the end of your day), revert your commits if you cannot make the integration succeed.

Frequent Commits

Efficiency gained by batching can be mere illusion. In most cases batching of work is not done for efficiency of current activity but to avoid inefficiency of following activity. In this context, commiters work for long time before committing because next activities like running the build, waiting for continuous integration build, takes a long time or is unreliable. It is the responsibility of good agile teams to address these issues. Size of code commit is a metric indicating how good the build and continuous integration is. If the build is fast and reliable, people would commit more often as no one likes holding onto changes locally. This is because if one has a lot of local changes one has keep merging them with everyone else till she commits. Once you commit it becomes part of common codebase by which not only there is no merge headache, but everyone can see the code that you have put in. Everyone being able to see your code is important because then they can provide you feedback, they can adopt to what you have done.

Agree on what do I do when the build is yellow?

It is pretty logical when one should commit given the state of continuous integration except when it is yellow. Teams should agree upon what to do in this state. Personally I lean towards a continuous integration system whose motivation is to allow people to commit, instead of being a police, hence my vote is for commit on yellow.