Note, these are not ‘sixteen simple steps’ – there is nothing ‘simple’ about this.
This piece is a brief introduction on how to implement a software metrics program. It is based on a real life experience.
Why would anyone want a software metrics program? Because it is the primary basis for any improvement in the software development process. Most software development produces far too many defects, and it takes far too long. If that is not the case where you are, congratulations. Tell me how you did it. I think the quality of our commercial software is unbelievably bad, and it takes far too long to get it to market.
If you want to improve your software development, this is one of the pre-requisites. As the great Yogi Bera once said (well, he didn’t actually say it): “You cannot improve what you cannot measure.”
On the other hand, metrics alone is not going to cut it. You need to motivate, organize, define process, find tools, build a culture for productivity, and 10 other things to really make this work. I’m just trying to help on the one I found the hardest – getting folks to count things!
The sequence of the steps is somewhat important – I recommend that you do them in this order. This first section gives you a brief overview of them all, so you can get a sense of how this works. The rest of this paper takes them one at a time, and shares some of the wisdom and pain my team discovered as we went through this process.
This is hard work, and it will take time. You didn’t learn to run the day you were born. You have to learn to walk, then run. You don’t learn every dance the first day of dance class – you learn the basic box, one step at a time. I’m no Fred Astair, but I had a teacher who showed me how to waltz, fox trot, tango, rumba, cha cha, swing – all, one step at a time. It is not rocket science, not brain surgery – it is even simpler than auto engine mechanics. It is just hard work. You can do it – one step at a time.
1. Be the boss.
2. Tell Stories.
3. Educate yourself.
4. Educate your staff.
5. Meet once a week
6. Produce a regular report.
7. Do a gross “cost of quality” measure.
8. Set an unbelievably high goal for quality.
9. Count every defect.
10. Assign a cause to every defect.
11. Analyze the cause of defects.
12. Establish quality improvement incentives.
13. Count your baseline product.
14. Count every change to the product.
15. Set an unbelievably high goal for productivity.
16. Analyze the development process step by step, and improve it.
17. Establish productivity improvement incentives.
(17 steps just doesn’t have the ring of 16 – sorry)
If you aren’t the boss, you are going to find this difficult to accept, but it just doesn’t work. I’ve tried it both ways, and I couldn’t get it to work unless I was the boss. The other side of that, you can’t get this to work by just telling people to do it – but sometimes you do need to just tell them to do it. I know that sounds contradictory, but life is messy, you know.
You can’t just mandate it. It’s very important to engage the staff and get them excited and interested in “doing metrics”. It has to be that everyone that is doing it – not you. Your role is cheerleader and coach. They are the players. Some of these steps, “cost of quality”, weekly meetings, reports and incentives, are aimed at bringing everyone into the game. Use the word “we”, not “I and you”. Tell them that this is our ‘salvation’. We are pressured all the time to do it faster, better, cheaper – this is one tool that will help us accomplish that goal. Our real goal is to work normal hours, have a balanced personal life, and still be the organization that generates the most code with the highest quality. I think you can do that. If you don’t know how to motivate your staff, to give them focus and goal and mission – then get someone to help you do that, or find another way to make a living! I’ll suggest some ways to do this as we go through the steps, but this is really the stuff of leadership – not management, necessarily, but leadership.
It’s also the best way to push back on schedule pressure. You know what schedule pressure is, don’t you? That’s when the customer says: “Great, we want all of that, but we can’t wait that long, and we can’t pay that much – do the same work, but faster and cheaper.” If your primary technique to ‘faster’ is to lean on the software developers, you have just created “schedule pressure”, and you have guaranteed that the defect rate will rise to new heights. The push back here is to show yourself and the customer the true cost of those defects. More on that later.
If you aren’t the boss, and you really want to do this – talk to the boss. Give her or him a copy of this article, get a couple of the simpler books, go to a class. Howard Rubin of Meta, and Capers Jones have been doing metrics a long time – they are both inspirational speakers. Ask them to come and introduce the idea to the boss and the management team.
If you are the boss, you are the key – you can “just do it”. You don’t have to run the whole division or department – you have to have responsibility for one software development entity.
If you have a management team, managers or project leaders that report to you, get them on board before engaging the staff. There’s nothing like having a large staff meeting where one of the managers starts debating the thing with you. I don’t mean that they have to agree absolutely – but they have to be on board. In fact, if you have a complex organization, you absolutely have to have the managers engaged to the point where they are actually doing it – not you. If that’s not possible, then do a pilot in one segment of the organization – something that is isolated enough that metrics and process changes can be done without affecting the rest of the world.
It’s up to you if you want to tell your boss and your customers about this effort – personally, I would get the thing up and running before I told anyone about it, outside of the staff involved in making it happen. There’s nothing like raising expectations and then have them crash to get yourself fired! There was this start up company formed some time ago. Things were going great after just a year, and the boss figured that the company bonuses would be on the order of a whole year’s salary for everyone. He let that news out of the bag in the third quarter. Well, between one thing and another, problems, etc., the bonus actually came out to be only half of a year’s salary. So when people were given this enormous amount of money, they all complained bitterly. Avoid the dashed expectations.
Learn to tell stories. I didn’t make up that story on the bonus, but I am not really familiar with the details – I heard a professor tell it once in a lecture. There is something magical about our brains and stories. People hold on to them better than just ideas. It’s like the difference between text and a song. It is just easier to remember, and we are more moved by the poetry and the song than we are the bare text. Stories work like that too. After a while, you will build your own stories – but I wouldn’t hesitate to make them up as needed. Analogies are almost as good as stories – people seem to really retain them. Hopefully you will find some useful ones in this text. I’ll try to provide some examples as we go.
There is nothing like knowing what you are doing, and learning from what other people have done. This is just a short list of sources and ideas on bringing your own background on this up to speed.
When I started compiling this list, I figured I would give you some choices, but half way through I changed my mind. Read all of these – don’t skip one – it’s a short list – you can do it! If you find a better one, let me know – thanks!
Quality is Free, by Phil Crosby. I started with this one many years ago, and found it magical. It’s in paperback – in fact you can probably find it in a used bookstore. It is an old book –and the origin of the original 16 steps as borrowed from AA. The book is not about software – but it can be applied to any type of business.
One of the things I discovered later on is that you do NOT want to go around telling people that '‘quality is free”, the Crosby mantra. Crosby is right – quality is more than free, it generates major returns – but no one believes that up front just because you say it. They don’t just laugh at you, they treat you like a crazy idealist. That is kind of fatal for motivation and credibility.
Crosby has another nice phrase, which is also very powerful, and also gets you in trouble if you use it. He advocates something called “zero defects” as a goal. I like it, and it motivates me. But I also found that people treat me like a nut case if I say it. The goal is zero defects – although we realize we will likely never get there. The idea is to raise the perspective that every defect is really a defect – every one. If we have a problem, we need to analyze it enough to figure out how to never have it again. Normally, people just fix problems – they don’t stop to figure out where they came from, and how to prevent them. The book is all about prevention! That’s the basis of the “zero defect attitude” – but it tends to turn people off until they get inside it.
PeopleWare, by Tim Lister and Tom DeMarco. This is a wonderful little book – also in paperback. The premise here is an experiment that Lister and DeMarco put together to assign the identical software development problem to some 200 different teams and then to measure the differences in productivity and quality. What they learned was amazing. Productivity is primarily driven by the way you treat people – not by the process or the language or the tools. They are all factors, but the essential one is human motivation. That’s a very simplistic synopsis of a wonderful little book. Tim Lister is a good speaker too. If you want to spend some big bucks, have him come and talk to your group. He tells a lot of stories!
Pattern Languages of Program Design, edited by Coplien and Schmidt. Jim Coplien (Cope), of Lucent, has an article in this volume that speaks to how highly productive projects work. He identifies types of “organizational patterns”. The article is “A Generative Development-Process Pattern Language”. I didn’t want to read it after I read the title, but it is well worthwhile. Jim studied the “interaction patterns” of a large number of software development projects, and he has identified some “patterns”, or “heuristics”, or “rules of thumb” for what works in a software project and what doesn’t. One of the most productive software projects he studied was a C++ project – not a language where one would expect high productivity and quality. What this piece teaches you is that you need to do things the right way to actually improve quality. Counting isn’t enough – but it is the basis for being able to tell if you are moving in the right direction. Cope’s organization patterns are also a bit of genius work – I think it helps a lot to know that this type of information even exists, and to hold them up as examples to a team trying to improve what they are about.
There are several guru’s of software metrics out there – I’ve had dealings with a few and found them very entertaining and enlightening. Sign up for one of their courses, engage them for consulting, whatever you can afford. Both of them respond to informal queries with considerable grace and speed, considering how much their time is normally worth on a consulting basis. They also both market their services through a consulting arrangement. I’ve not had the pleasure of being a customer, but if you have some commitment and dollars, I’m sure either of them could help immensely.
Capers Jones – Chairman, Software Productivity Research – www.spr.com. Capers understands function points, and knows that they are more reliable, etc. But he is a pragmatist when it comes to implementing something, and is not above using lines of code if needed. He has very good statistics on specific projects, types of programming languages, etc. If you can’t afford to go to one of his sessions, buy a couple of his books.
Howard Rubin – www.hrubin.com. Howard has a bit of a research and academic background, but he has lots of experience with real world problems, and is also very entertaining. Howard has a mountain of data on software productivity and other things that he gathers every year as part of a worldwide benchmark. He’ll share the data if you ask, and also ask you to take part in the survey. Howard offers his wisdom primarily through Meta’s quality assurance consulting.
As I’ll say more than once, this is tough stuff, hard work. You really need a support group – people of a similar bent and ilk, who can share their experiences, and can help keep you motivated. Think of it like joining a church – those religious values are tough to hold on to in the day-to-day battle – the same is true of software metrics. You need a place to go back to that will reassure you that the battle is worth it.
There are several software metrics organizations out there. I found that the SPC, Software Productivity Consortium, was very helpful. If you get into function points, the International Function Point Users’ Group (IFPUG) would be very important for training and standards. If funds are really scarce, form your own support group where you are. There have to be local practioners out there who are also interested in the same thing – find out who they are and invite them to start a cooperative group. Talk to your sales types – hardware, software, consulting. Tell them you are interested in starting up a group like this, and ask them for some names of likely candidates. They network all the time, and they know everyone. They can help you find people who have similar interests.
Invite the group to your place, serve them food, have them draft up a mission, and first steps to get up and running – and start telling your stories. Think of it as your monthly support group, and sharing stories is what you are about.
OK, you have a glimmer of an idea of what you want to accomplish. Now you need to get some training into the hands of the folks who will actually do this.
The resources named under “educate yourself” are in this category. Jones, Rubin, the books and their authors – those folk can move a group from apathy to excitement. If you can’t get the actual people in, start a book club and give everyone a copy of the books, and start reading! Use the weekly meeting for exploring the ideas and how WEmight implement them. There are precious few articles on software metrics published – when you find one, point everyone to it. Build a DB of metrics resources that everyone can reference.
Along the way, staff will need specific tools and training to help them do this. Courses are available on metrics in general, function points, software inspections, process improvement, methodology, software maturity continuum out of Carnegie-Mellon, etc. The more clearly people understand the power of this stuff, and how other people are using it, the more they will be encouraged to keep at it.
This is hard work. You just can’t do it in a haphazard fashion, and if you don’t do it all the time – it will simply go away. It’s a lot like religious values. We know that these are important, that they are part of the foundation of our way of life and of our personal sense of meaning in life. But if we don’t pay attention to them on a regular basis – they just kind of wither away. If you really want to do this metrics and quality and productivity thing, you have to do it at least once a week. Think of it as “going to church” – set up a ritual, formality, regular meeting, agenda, etc.
Try to generate some excitement around the first meeting. Serve food and drink, invite everyone to one large meeting. I think it is important to have everyone there for the kickoff – just get a bigger room! Jim Coplien has a pattern name for this – “all in one room”.
If you have the resources to do a “workout”, do one around software quality and productivity improvements, and set up the weekly meeting as the follow up. If you are unfamiliar with a workout, just gather everyone together and put the question to them – what can we do to improve our software productivity and quality in a major way – both modest steps, as well as big time stuff. Split them up into small groups, have them brainstorm for an hour, and ask them to report on the results as recommendations. And whatever they say, accept them, affirm them, commit to follow up on them. Then establish the weekly meeting as the follow up mechanism. Put their ideas on the first few agendas, right after mission and meeting rules.
You have to have a mission, and it doesn’t need to take forever to draft it. It gives people focus, keeps them on track. If you have trained meeting facilitators, use one for this. If not, have everyone at your first meeting help build it. Get them all to think about how they would word the mission statement for this group, and have them write that down. Then go around and get everyone’s ideas out – put them up on a flip chart. Then wordsmith that until you get consensus – this is not about majority rules – we want everyone on board here, even if grudgingly.
Try to keep it short – if it gets too long, then go to a “mission statement” and “list of objectives”. The mission is the overall goal, plus a list of things we want to accomplish.
Publish it, give a copy to everyone, make it into a poster or banner to post at the front of the room. Don’t be afraid to review the mission, freshen it, explain it. The goal here is to get everyone engaged – not to draft a document. But the document is handy as a goal and as a reminder. When you get new players, you inevitably have to go back and engage them – this gives you a short hand way to do that.
Every regular meeting needs rules. Brainstorm this again with the people in the room – propose some for consideration, and come back to them for the first few meetings to see if they work. They will have good ideas on how meetings work, but here’s a list of suggested ones:
1. Attendance – it’s ok to miss a meeting. The published minutes will bring you up to speed.
2. Timeliness – it’s very important to start the meeting on time. If you really can’t get there on time, we can live with it, but make every effort. The meeting will start precisely on time.
3. Length of meeting – the meeting will also end promptly, or even early, if we get the agenda completed. This again respects people’s time.
4. Agenda – we have one, and we stick to it, more or less.
If important items are raised that are not on the agenda, we put them on a “parking lot”, and get to them at the end of the meeting, or we put them on next week’s agenda. If this is too hard, try building an agenda at the start of the meeting, and prioritize the discussion based on the topics raised by the group.
5. Chair – we rotate the meeting chair. The chair has the authority to focus the discussion, keep it on the agenda. The chair is also timekeeper – we need to move on to the next item, and we’ll come back to this one at the end or at the next meeting.
6. Minutes – there are minutes from every meeting, so someone needs to take them. These are powerful reminders and to-do lists for the attendees. And they help bring non-attendees up to speed.
7. Action items – every meeting results in some action items, noted in the minutes.
8. New members – we welcome new members and infrequent attendees – but they must go through an orientation, and not during the actual meeting. They are given a copy of the mission and objectives, and prior meeting minutes, so they can get up to speed without distracting everyone else. I found it terribly disruptive to try to orient a new player during the meeting.
You get the idea – the group will have good ideas also.
I think this is critical. It is a visible symbol that everyone owns this process and result – not just the boss. After the group is well formed, and mature, and able to interact freely, then they may decide that one or other member is really good at running meetings – a natural meeting facilitator, if you will. They can name that lucky person permanent chair!
As the boss, you should clearly attend every possible meeting. But you should also keep your mouth shut for the most part. Restrain yourself. I know you have good ideas and good experience to draw on, but wait until someone asks you for it. With authority and salary control on your side, you hold immense power to direct or distract the group. They will end up thinking you own the thing, and not them. You can try to facilitate the meeting, even if you are not the chair. Reflect what people say, make positive comments on ideas put forth, encourage discussion – “what do the rest of you think” – things like that. Don’t do too much – a trained meeting facilitator would be better if you have one.
After things are up and running, make sure you miss some meetings, and see what happens. It will help you understand if the group is mature enough to bear your comments.
After the group is really mature, and accepts you just as a member, you might take the risk of putting forth one of your ideas or evaluations. Hopefully by then, the team will be able to reject it if it doesn’t fit.
This does not mean that you abdicate leadership – far from it. It’s just that you lead in a very different way than you are accustomed to. You suggest, you raise perspectives, you generate ideas, you challenge, you help. Hold the mental image of the basketball coach. Basketball is too complex to have set plays run all the time – the players have to be good, and have to respond to the initiative that is in front of them. The coach gets them into condition, gives them training, holds up their confidence, and encourages them to respond when the opportunity appears.
Along these lines, you don’t have staff – you work with people. People don’t work for you, they work with you for the customers. Change your language. People only give their best when you treat them as volunteers. Get rid of the idea that you can actually tell anyone what to do. You can coach them, help them, encourage them. Once you are reduced to telling them, you are running it from there on.
It is very helpful to have an agenda set ahead of time, so that team members know what to expect, what to bring to the meeting, etc. This is very difficult to do. As an alternative, create the agenda at the start of every meeting. The chair gets up, walks to the white board or flip chart, and asks for agenda items, then jots them down. If there is an item planned for this meeting, put it at the head of the agenda. Anyone can create an item like that by raising it at a prior meeting, or notifying everyone before the meeting by e-mail, that they would like this topic discussed. That guarantees it a place at the head of the agenda. That works well enough, that people will actually remember to do it.
The chair then decides which items should go first, with the input and consensus of the group. “Let’s start with x, and then do y, and then the rest of the stuff as written. OK?” This works very well. It gets the group focused at the beginning, tells them the meeting is starting, outlines the topics, and gives a sense of when we will be done. I found this dynamic agenda to be such a great idea that I introduce it to every meeting I attend.
The group has to have a sense that they are doing work – they need tangible action items that are accomplished, and reported to the group. In the beginning, crafting meeting rules, follow up items on the kick off meeting ideas, etc., should be assigned to volunteers. As you get into the nuts and bolts of the next steps below, team members should take those on, defining procedures, meeting with people, preparing reports. If you have someone who is a numbers type, runs Excel or other reporting products, suggest that they pull together some report “mockups” – if we had data, what would it look like. What kind of data do we think is useful? Propose reports, mock up samples – even before you know how to collect that data.
The meeting has to generate minutes. These are reminders for the action items, progress reports, and they help new members get up to speed. The chair should ask for a meeting secretary volunteer every meeting – or the group can decide to rotate the chore by a set list. Store the minutes in a public place – like a discussion DB or website. Make them searchable. Keep the format consistent. It’s important to note attendance every week – even if just a list of names. I tried keeping a kind of calendar of attendance for every regular member and repeating that every week – the idea is to encourage attendance!
The initial ideas are hot, and you should act on them while the group has enthusiasm and energy. But at some point, the group needs a plan. Do a brainstorm, ask for ideas, and lay out a simple set of steps that we think we should accomplish and the order we should do them in. Write it down, discuss it. Put a gross time line on the plan – what does the group think it will take to accomplish this item – what effort, what target date. It can always be adjusted, but it really helps give focus and order to what is going on. Otherwise people may have a sense that they have been working hard, but don’t really see that they are accomplishing very much. This gives them a sense of progress, history, accomplishment, and a vision as to where they want to go. It is very powerful.
I think once every three months, the group should stop and assess where it stands. I’m assuming that this is a large group, and it is going to take years to get this thing done. If that is not the case, and this will be done in 6 months or something, then do an assessment monthly.
Again, assuming a long-term effort – multiple years – it pays to assess the plan annually. Review the mission, objectives, meeting rules, plan, and draft a new one. Do it in another “workout” with food and entertainment – keep it fun and interesting.
You already have weekly meeting minutes – why a report? I think it gives you a chance to focus the group, and review what has been done. It also gives people outside of the group a more structured way to understand what is happening. This is where you continue to raise the vision. You can also point to other examples, training resources, opportunities. Make it short, make it visual, put charts and graphs on it as they become available. Keep the thing in front of people – it’s another way to get the attention of those who don’t “come to church” every week!
As you get some measures, post them visibly on the wall somewhere as another reminder of what we are trying to accomplish. Think of it as a visible icon of our commitment.
OK, most of stuff up front was preparatory. This one is a real product, with a real message. The Crosby book will explain this in detail, but in simplistic terms, this is the cost of all of the systems repair work caused by defects, as well as the prevention efforts. A full cost of quality will also include the downstream impact on customers and their productivity, but that is really hard to do. Typically, the normal ‘cost of quality’ runs between 40 and 60 percent of total systems effort. Think of that – half of your systems dollars are spent on finding, fixing, rerunning things that are wrong. What if you could reduce that to 5 or 10%, a very doable goal, by the way. Think what that would free up in terms of resources or dollars for the corporation. This is typically a very, very telling report, and has a lot of motivating force.
This is not only a good thing to do, but doing it will give you a sense of what kind of measures you already have in place. For example, you will need to capture effort hours and equivalent costs for systems work. If that doesn’t exist, you have some real background work to do.
You can build a cost of quality report in pieces. Initially, just start with the numbers you currently have. Surely you are capturing work effort somewhere. Categorize that work effort as to whether it is involved in defect detection and repair, and reruns. That’s a good basic start. If you don’t have fully burdened charge back, estimate what that is. If you are not yet capturing the effort numbers, ask a group of people to independently estimate what they think the defect discovery and repair and rerun effort is. This is called the Delphi estimating technique – ask a bunch of folk and average out the answers.
If your first one is based on estimates, set out to build the metrics needed to do this. This is your gross speedometer, as it were.
This is telling stories. There are some good studies out there on what other people are doing. Howard Rubin and Capers Jones have some interesting numbers. Howard says that 10 years ago, the productivity spread between the lowest performing company in his survey and the top performer was about one order of magnitude – the best ones were 10 times better. Rubin says that in recent surveys, the spread is now 200 times! Once you have your number, defect insertion rate, productivity per person month, whatever – compare it to some of these studies. If you are doing a good job, you should find yourself in the middle of the pack – hey, most of us really are average – that’s what average means! But we want to compare ourselves to the leaders – and someone turning out 200 times the code we are, with higher quality, is worth studying to see how they do it.
I also used another technique. I would question an individual, a group, the boss, on what they thought was the potential improvement in our organization. Could we improve 10%? 20%? Typically, people give numbers in that range. Then I would point them to companies doing 200% better than us, and ask them to raise their goals. The long-term goal has to be “world class quality”.
That is not to say that one can get there all at once. An unbelievably high goal in the short run will just discourage people. Tell stories about the long-term goal, but emphasize that we are doing this one step at a time, one year at a time. Set a more realistic goal for each year – say 10% a year. I found that people could achieve a 10% improvement in error reduction or discovery, just by paying more attention. But they can’t do that year in and year out at a 10% annual clip. That kind of improvement requires a fundamental process change.
I can’t emphasize this one enough – count EVERY defect. You will get some push back on this: “Some of these things are so minor they can’t possibly be included in our defect count.” “It takes more resources to count them than it does to fix them.”
Be strong, hang in there. Defects should be categorized by severity, so adopt a 3 or 4 level defect severity rating. Allow people to assign the really trivial defects to that category – “trivial”. The others might be “medium”, and “major”. Do the criteria for assigning them any way you want – by impact, work to correct, or just general gut feel. Just capture them all – you can always reassign them later.
When I say count, I mean capture. Every defect has to have a record made of it with some minimal information. Put these into a simple on-line system so people can enter them at their desk as they work on them. Make it an Access DB if need be. At a minimum, include:
- description,
- severity,
-cost to repair.
You will find more things to capture as you begin to analyze them. Some others might be:
- release where defect was inserted
- phase of development were defect was discovered
- phase of development where defect was inserted
- component containing the defect.
When people complained about having to capture every defect, I told them the story of the spaghetti junction interchange in St. Paul. US Interstate 94 goes right through the heart of St. Paul. The road does an S curve in the middle of downtown – probably to avoid some old buildings and churches, and to follow the topography. The way this set of exits and entrances was designed, the focus was on enabling people to get on or off the freeway with the most safety. People would exit left, enter left, right, whatever worked. The S curve also interconnected with Interstate 35E – another major problem. Well, the designers fixed it for those getting on and off, but made it a nightmare for those trying to stay on. Because lanes kept moving, appearing and disappearing, it was impossible for anyone to get through the S curve without having to change lanes many times.
The original designers clearly thought that they had a good design. Fortunately, the Minnesota Dept. of Transportation collects accident statistics. They found that this intersection, this “spaghetti junction”, had more accidents per volume of traffic than any other piece of highway in the state. Taken individually, each accident was clearly the result of someone not paying sufficient attention. But lumped together, it clearly pointed to a major design flaw.
Thanks to that data, the interchange received funding for a major upgrade. All 3 lanes now go entirely through the city – there is no need to change lanes whatsoever. It was so successful that they went over to Minneapolis and made the same changes.
In the systems arena, when we first started to capture all defects, we found we had a fairly big pile of trivial errors. None of them were major, but the total number was kind of surprising. They were consuming quite a large piece of our work effort. So we sat a group of people down and asked them to analyze them and find a way to reduce them. They came up with a variety of ideas, from spell checking documents, to better inspection processes. The number of trivial errors melted quickly.
People won’t like this one either, but stick to your guns – it’s important. There is almost an exact correspondence to defects found in unit test and those found later in production. You will typically get more in unit test, but, unless you change the people or the process, you can use the defect rate at the unit test level to predict how many should be found in quality assurance testing, and in production.
One technique I have read about is to separate the responsibility for program design and coding from the unit test. Typically, the programmer does the unit test on the code they designed and wrote. If you separate that responsibility, have the testing done either by a different programmer, or by a QA professional, it tends to raise the standards of the programmer. When I did programming for a living, I always figured I would find problems in testing – I did almost nothing to prevent problems. The best programmer I ever had the privilege to work with, would always return a first compile with NO errors. I had never heard of such a thing. I thought he was wasting his time catching errors that the compiler could find more economically. But he also had the distinction of having NO run time errors! He just paid an extraordinary amount of attention to doing it right the first time. I am not sure that this is something one teaches – it may be something one is born with. If the latter is true, you need to find the folks on your staff that are like me – I would always figure it out later – and have them do something other than coding. I never even calculated the page break line count – I figured the first test would show me how far off it was!
There is some interaction here with the amount of work effort or product. As the work effort for a single release grows, so does the defect insertion rate. More complex work, extending over a longer span, gives rise to more errors. The corollary of this is that one can improve quality greatly by doing all work in small chunks. Monthly releases are very doable, while weekly would be a tremendous goal. This means that some changes have to be broken down and implemented over the course of several releases – this is a good thing, not a bad outcome.
Extreme Programming has introduced a whole new wrinkle to defect counting and testing. This method promotes the idea of writing automatic unit tests before writing the code. This way, all errors are detected instantly. When someone changes the code, they have to pass all of the automatic tests before they can install it in the system. I think this is a brilliant idea, but not many tools allow for this kind of testing as yet. Even with automated unit testing, many other errors will continue to be inserted, so we still need to count them.
Every defect has to have some kind of cause assigned. Generally, these can be assigned right up front when it is fixed. In some more difficult cases, or more sensitive ones, some discussion will be needed to assign the cause. Whatever it takes, everything has a cause. It is a good idea to assign some standard causes, with room for textual explanation of the cause in more detail. They are all caused by human error – we know that – but we need to know how they got by the defect detection steps. A simple standard set might include:
- coding error
- incomplete test plan
- missing requirement.
There is a more complete list in the appendix.
Once a month, or at least once a quarter, sit your lead analysts and project leaders down in a meeting with the data gathered so far. Ask them to look at all of the problems of the last quarter, and the trend of problems over the longer perspective. What does it tell us? Is there any way to categorize some of these problems and come up with a way to get rid of them, or at least to catch them earlier? The earlier a problem is found, the less costly it is in terms of cost of quality.
If this seems overwhelming – you have way too many problems – then start with the more serious ones found in production and QA testing. As you progress, move earlier in the development cycle, because the earlier problems are generally caused by the same things, and catching them earlier gives a much higher return.
If coding errors are the dominant thing (they usually are at the beginning), then implement code inspections. This is a formal process where a group of peers with a leader walk through the code and identify errors and suggest improvements. There should be minutes from these inspections, and the defects identified should also be captured and categorized. Use some formal training to get people started on this – it helps avoid sensitivity to having others look at my work. When I was programming I never let anyone see my code. That has to change.
The extreme programming people use something called pair programming. It’s not easy to implement, but it does seem to work to get a public attitude about code, and the productivity actually goes up, even though 2 people are working together all the time.
OK, now we know what our cost of quality is, we know how many defects we are inserting normally, and we have some ability to predict the number of defects we will find in production. It’s time to set some improvement incentives. One way to do this is to figure out what a 10% improvement in overall quality is worth to the organization. With your cost of quality, you have a shot at that.
If the improvement target is attained, then give half of that gain back to the staff as a one-time incentive. Start each year fresh – put last year’s gain in the bank, and set a new target. This will literally pay for itself many times over. Don’t be afraid of a big payout number – remember what you bought with it. Big dollars do actually motivate most people – it’s why capitalism reigns.
Typically, incentives like this should be shared across a group that is working together. Separate software projects that have no interaction can have separate goals – but any group working on phases of the same thing should share the same incentive. Don’t forget the clerical help and programmer assistants, etc. They can do a lot to solve problems, and they can bring a certain public pressure on the other staff to do a better job.
At this point, you have a handle on your defects. Now to the hard part, how to count the product. First of all, why count the product at all? We are doing good, defects are falling, we are saving money, etc.
One must count product because errors and productivity are directly related. There is no faster way to reduce errors than to stop making changes. Taking twice as long to do changes is also a possible technique. Testing things to death is another well-known method. Each of those has a very negative impact on productivity.
On the other side, pushing more work into a given release has good productivity results, but is very damaging to quality. The same is true for overtime – it looks like more gets done, but the quality falls so much that there is less actual product delivered.
The result – one must be able to prove to customers, and management, and the staff themselves, that productivity is also improving. The good thing about this is that reducing errors is almost always the easiest thing to do in order to increase productivity.
I say count the baseline product, because you need to start with some understanding of what you have. Function points are the ideal method here – with proper training, an expert can replicate a count within 10%. The problem with the technique is that there are some judgment variables in the calculations related to program complexity. When financial types see these variables, they typically have a fit. I would tell them not to worry, because the judgment calls used in determining fixed assets and ROE are a whole lot more variable than those used in Function Points.
One can count the base case – the number of function points found in a system, module by module. This gives a base line, for things like maintenance effort per function point, errors per function point, etc.
If function points get to be religious wars, switch to lines of code. Capers Jones has some algorithms for counting lines of code that can be translated into function point equivalents for a given language and technical environment. The good thing about this is that most source code management systems can support lines of code counting – even with user-supplied algorithms. The genius here is not to tell anyone exactly what is going on. Show productivity points or work points as the measure – not lines of code. It is entirely possible to generate dead code or redundant code that appears to add to productivity – it is almost impossible to generate unused function points.
We never found anyone padding the count, but I think it is pretty clear what action one has to take if things are not always honestly compiled. If we don’t have integrity in our internal numbers, do we have integrity in anything?
After the base line count is completed, it is important to count every change. Every change is work effort – whether it is inserting, modifying, or removing function points (or lines of code). Each task should be estimated as to work effort, and as to function points affected. With enough experience, the function point estimate can drive the work effort estimate.
I collected numbers once on how much the original estimate of work for a given change departed from the final effort. There were a few projects where the estimate was high, but for the vast majority, all estimates were significantly below what the actual work required. People are eternal optimists about what they know, what they can accomplish, and they assume that nothing will go wrong. In fact, something always goes wrong, and they always run into something unexpected. After some experience, one applies a contingency factor, and possibly, a change control factor to every estimate. Contingency is to cover those things we didn’t know. Change control is to cover changes in the business or those things requested by the user between the time we started working on this and we finished. We used to use 10% for contingency – an extremely low number – and 40% for changes.
Once the work effort for a release is in hand, one can also predict the defects, by using the known defect insertion rate – number of errors inserted per function point. Knowing how the discovery of these is spread through the life cycle, one can also predict the number that will be found in phases. Imagine being able to tell the product manager how many errors would be shipped in production if we stopped QA right now and shipped the product.
This is the same as for quality. There are shops out there that are doing a great job – we need to look to them for leadership and inspiration. As with quality, attack the goal a step at a time, year by year in doable chunks.
This is the same as for defects, and should have similar results. Quarterly, at least, sit the best people down and ask them how things can be improved. What have they seen others do? What ideas do they have? What tools, techniques, process changes might help?
Reuse and component based development has the best hope of improving both software quality and productivity. But it takes a fair amount of commitment to build the infrastructure necessary to support this approach. If no one sees any potential benefit in hard dollars, building reusable components will simply not happen. Capers Jones found that the highest productivity he had seen was attained by a classic COBOL shop that had a rigorous, formal program of code reuse.
This is the same as for quality metrics – make them gain sharing, based on the group effort, and plan to give away a sizeable amount of money initially. As savings get captured and institutionalized, the returns will be compounded.
Assuming you achieve the goal of a highly productive shop that is producing world-class code with extremely low error rates, I don’t know what one does for an encore! At some point, the judgment has to be that this is working fine, and it is time to focus elsewhere. The risk is that things will fall back to zero if the focus is not kept high. I would continue the weekly meetings; I would set lower incentive targets as one approaches very high levels of quality and productivity, and establish a kind of maintenance environment.
Whew – there you have it – a recipe for a highly productive software development environment. It’s a bit more complicated than chocolate chip cookies, but you can do it – others have done it and prospered – you can too.
A - REQUIREMENTS CHANGE
If changed requirement was written on the work order request.
B - MISSED REQUIREMENT
If the requirement could not be found as part of the original requirement document.
C - MISINTERPRETED REQUIREMENT
- Did not understand what was in the law or regulation change
- Mis-read the requested change
(note: This error can be caused by a non-IS person, where the wrong specifications are delivered).
D - DESIGN CHANGE
- Customer didn't like the look and/or feel.
- The design did not conform to requirements.
E - MISSED DESIGN
- An item was not in the design paper.
- Error handling not understood.
- Efficiency problems with design
F - MISINTERPRETED DESIGN
- IS determined the function would not work properly as designed.
G - CODING ERROR
-Program does not conform to design.
- Errors of omission fall in here - errors such as missing period.
H - INCOMPLETE CODING
- Logic errors
- Loading/reading a table and not checking for the end of the table
I - LINKAGE PROBLEM
J - INITIALIZATION PROBLEM
- Failing to initialize a record, field or segment properly
K - DS CONVERSION PROBLEM
L - TABLE PROBLEM
Any problem with a rate table
M - DATE TUNING PROBLEM
N - FORMS TABLE PROBLEM
O - DUAL MAINTENANCE PROBLEM
P - WRONG VERSION
Q- NOT A PROBLEM
Problem researched and it is determined the system works properly. This may be a sign the customer needs training in some area of the system.