Introduction

This was the article which introduced me to Continuous Integration. I first read this around 9 years back. I have practiced continuous integration in all of my projects since then. The article is good starting point if you are new to it.

In our context this means:

Integration: Putting the code of entire team together and building it to find out whether it passes defined build.

Continuous: Doing it as continuously as and when check-ins are done.

Goals

There are two key aspects of a development project, which depend heavily on continuous integration, forming its goal:

Ensure good health of the codebase

Developers in a team define criteria for good health of the codebase. These criteria differ from one team to other. Some team might define it to be successfully compiled and unit tested code, while other team might choose a ready for deployment state as this criteria. While everyone would want the later, they might not choose to keep this contract every commit of the code. Whatever they choose Continuous integration should prove whether this contract is adhered by all developers in a team or not. This contract is influenced by multiple factors like performance of build, testability of codebase, skill of the team members, etc. Once the contract is arrived at, good agile team respects it and continuous integration in a way just provides the information about commitment of developers to the contract.

Ensure seamless flow between development and downstream activities (testing, deployment)

Most teams stop at the first goal. But continuous integration should also ensure that its output is valuable for downstream activities. It should enable useful flow between implementation and downstream activities. Simply stating, as an example, a green integration build should guarantee that testing team can use its output meaningfully. A green build is meaningless if a tester cannot test the story.

Deliverable of continuous integration

Feedback to the commiter about status of her commit

Artifacts for downstream activities

Success parameters of continuous integration

Performance

Response time of an integration run is arguably the most important parameter. A really long build time can possibly single handedly guarantee the failure of a project. It is important to re-iterate that build time is time taken for integration to complete the hand over to next manual downstream activity. If the build is broken up in multiple steps, the build time is equal to time taken by all steps before manual step kicks in.

Correctness

This really is about whether there are false negatives and positives in the build. A lot of people who adapt continuous integration suffer from the problem of false negatives. When the integration fails for computing environment reasons, non-repeatable build scripts and timing issues; the commiters loose faith in it. This gives rise to a lot of bad practices like forcing of integration build, commenting/disabling parts of the build etc. This puts the team on a slippery slope of bad practices. In my experience the relationship between commiter and build system, is very much a trust relationship. Commiter's generally rely on their own intelligence than on in-correct build system, especially if they have not written it ;-).

Verifiable in commiter's environment

Continuous integration should use exactly the same build process as the integration build. In absence of such a setup, commiters would not be absolutely sure about the implications, when they commit. The additional steps which are performed only on the CI build should be really stable and runnable by commiter when desired. e.g. the CI build process might generate installers while the developers might not do so by default, every time they commit. Any gap between integration build and commiter's build should be evaluated carefully and decision made considering the whole team.

Continuous Integration and Continuous Automated Testing

Most agile team have different classes of automated tests. A set of tests are written by developers which are present along with the source code. Agile team which does test driven development has a lot of these. These tests vary in their scope but most of them are white box tests of some kind. They are intended towards providing faster feedback. Since they are not black box tests they don't guarantee correctness. Then there are other class of tests which are black box tests. These tests do not depend on any source code elements. They typically are long running but guarantee correctness of the tested scenario, from user perspective. The objective of these tests can also be to assert performance of the system. When these tests take very long, developers may not run them every time they commit.

It is important to differentiate between these two, as both of them serve different purpose. The tools used for implementing these might be same. There are two conflicting requirements from the automated programs. While on one hand we want to cover as much as possible, on the other they should provide the feedback fast (sometimes both can be achieved, but that's an exception). The pragmatic answer always is to draw the line some where in between, dividing the process and staging the parts separately. The part which isn't expected to work every commit act as continuous automated testing, the feedback of which is used differently.