TDD with Acceptance Tests and Unit Tests

Sebastian Kübeck 34 minutes later:

• • There’s actually more to do for QA people than extending acceptance tests to expose bugs:

  • Write Load Tests (and make sure that they’re run periodically) to make sure that the performance requirements are met. It’s also one of the few possibilities to expose concurrency issues.

  • Write Security Tests: Tests that mimic an intruder that attacks potential security holes (e.g. injection, XSS attacks etc.).

Christopher Gardner about 3 hours later:

• • James Shore has an interesting article favoring Fit as an analysis and communication tool over an executable specification.

  • http://www.jamesshore.com/Blog/Five-Ways-to-Misuse-Fit.html

David Chelimsky about 4 hours later:

• • I agree that manual testing as your sole source of testing is immoral.

  • I think, however, that manual exploratory testing is a great way to expose holes in automated test suites.

  • WDYT?

unclebob about 5 hours later:

• • Agreed. (As I said in the original article ;-)

Dean Wampler about 5 hours later:

• • I like to point out that the “redundant” coverage from unit and acceptance tests is like the security philosophy of defense in depth. You’re safer if you’re not relying on just one “thing”.

  • That’s not the primary reason for doing the two kinds of testing, of course, but it is an important benefit.

  • IMHO, the role of QA as the driver of the acceptance tests isn’t getting as much attention as it deserves. Too many organizations that are adopting the developer practices like test-driven unit tests still have old-style requirements documents and “passive” QA teams that only get involved at the end of the process. QA has always been the natural advocate between the customer’s requirements and the technology that implements them. QA-driven ATs really emphasize that role!

  • Finally, a particular kind of load test worth mentioning is the “smoke test”, where the system is run under a simulated load, hopefully on a near-production configuration, for some multiple of days to see what melts down.

David Chelimsky about 8 hours later:

• • Bob – I did read that, but lost it in the shadows of “Manual testing is immoral,” which reads (to me) as all-encompassing.

  • Thanks for the clarification.

Peter Wood about 11 hours later:

• • @Dean Wampler

  • I take ‘smoke tests’ to be a basic, small, base line set of tests which, if they can’t be run, indicate that no further tests need to be run until some serious issues are sorted out. e.g. the environment isn’t set up correctly.

Ben Simo about 15 hours later:

• • Bob,

  • You had me until “Manual testing is immoral.” This is wrong—and it doesn’t jive with the previous paragraph about exploratory testing.

  • Scripted manual testing that is high stress and tedious may be immoral. But including no manual testing may also be immoral.

  • People and machines have different strengths and weaknesses. Good manual testing does what people do better than computers. Good automated testing does what computers do better than people. Making people work like machines or expecting machines to think and create like people is wrong.

  • Manual testing is error prone, but so is automation—because it is developed by us fallible humans. A human tester may make a mistake once and it impacts a single test execution. A human test coded may make a mistake once at it will impact all executions of that test until the mistake is recognized and fixed. Maybe we should practice TDD for developing our TDD tests, and TDD for the tests that test our tests, and … :P

unclebob 1 day later:

• • Ben,

  • Correction accepted. Scripted manual testing is immoral.

Gary Williams 2 days later:

• • I have to disagree to some extent. In order for the business user to accept that the tests written actually reflect reality, the tests themselves must be tested. So automated tests aren’t, by themselves, sufficient proof that the application is correct and meets the requirements, especially if coded by the same developers. They do facilitate regression and, once the application has passed initial testing, make any retesting trivial. I have been bitten by developers making hidden (and different) assumptions than the testers about what something means resulting in a green test when things are still wrong.

  • Of course, this would not be a case where there would be a script anyway, since presumably a new/changed feature would need a new/changed script.

Dean Wampler 2 days later:

• • @Peter,

  • You’re right. The term “bake test” is a better term for the kind of test I was describing. “Smoke test” is a better term for the fast tests that find obvious problems quickly, as you said.

Joe Ocampo 8 days later:

• • >Programmers use the acceptance tests as requirements. They read those tests to find out what their stories are really supposed to do.

  • I couldn’t agree with you more. We recently began the NBehave project to address this very issue. It’s focus is to drive the point that in order for a story to be accepted by the development group there must be accompanying acceptance test.

  • http://www.codeplex.com/NBehave

Elisabeth Hendrickson 10 days later:

• • Fabulous essay! I like Ben’s rephrase to “Scripted manual testing is immoral.” But I’d actually prefer to see, “Manual regression testing is immoral.” If scripted, it’s immoral for the originally stated reasons. If unscripted, it’s immoral because it’s not going to provide enough information, fast enough, to support Agility.

Hans-Eric Grönlund 13 days later:

• • Great post, the best I’ve read on the subject.

  • In my experience it’s difficult to get business people do programming, even in a high level language such as FitNesse. Therefore, my development team usually helps in the process of designing the acceptance tests, then turn them into automated tests using the same framework as our unit-tests. That process has lots of great side-effects, and it’s our assurance of quality.

Gary Murchison 20 days later:

• • So what is the outcome of the outcome of the unspecified / underspecified areas of the system that the “creative” tests expose. Should these be tackled (with new automated acceptance tests) within the same iteration or deferred until a later iteration?

Diego Sacchetto 20 days later:

• • One of the differences that Uncle Bob raised is between unit and acceptance testing. They both are automated tests. Manual testing falls under a completely different category and methodology. Agile promotes automation, after all we are in the software business (which is also an automation tool on its own) therefore everything should be automated as much as possible. How good we are to provide full automation and full coverage for requirements (acceptance tests) and functional logic (unit tests) will determine how much agile we are. The more we craft the art of automation the more we will replace the manual testing. As humans we don’t like boring repeated tasks. Think how many times on a straight road with a speed limit, you start your cruise control, simply because it is to boring holding the accelerator and not steering. Moreover people on the robotic industry are trying to design self-driven cars which purpose is to reduce car accidents by eliminating human errors. I do believe manual testing could be removed almost completely no matter the application domain. People are investing a lot of times and energy to provide frameworks to test anything (in the business I work: digital tv applications – we can automate the actions of a user that is using a remote control. Pretty impressive I would say). To achieve this goal we must first change the traditional meaning of testing. With Agile (TDD in particular) testing becomes a software development methodology. Therefore, traditionally, a QA person didn’t need to know to much about programming, but today to write acceptance tests you do need to know how to program and often in the same language that the application is written. But if you follow agile practices then the collaboration between QA and developers is a natural fact. Let’s no forget that the ultimate benefit of automation is more appreciated if we introduce the XP practice of continuous integration. I agree with Uncle Bob. As humans is more fascinating to study strategic way to work smarter rather then repeating over and over manual tasks. Keeping people self-motivated is the key, and for sure manual testing doesn’t do that.

amyguita1@yahoo.com 11 months later:

• • I have a question for you. Should developers regression tests their unit tests?

  • Thank you.

joe 11 months later:

• • When it comes to specifying requirements as acceptance tests, does anyone even see a need for formal use case or requirements documents?

  • To me, it seems like if you are truly attempting to adopt a leaner process, then creating the test cases should be enough of a specification.

  • My company is trying to create formal use case documents that tend to be difficult to follow and use that as a specification for development. I think that creating the test cases would be a more valuable artifact. Creating the cases helps elicit requirements from the business, aids in analysis, delivers an automated checklist for developers to follow, builds up your regression suite and essentially creates an artifact that serves as a record of what was built.

  • Of course, good analysis would go into the creation of the test cases, so that they accurately reflect what the business wants, but this analysis doesn’t necessarily need to be driven by formal use case documents.

  • Thoughts?

victor about 1 year later:

• • Your immorality claim on manual software testing is absurd. Software creation can also be high stress, tedious, and is obviously error prone. Yet despite this you claim that it is what humans do best. Clearly those aren’t the reason why manual testing is immoral or software creation would in turn be immoral.

  • The only unique value claim that you make is that it is wrong to turn humans into machines, but you have no justification for this. By this reasoning most manufactured goods are also created immorally since sewing clothes or crafting vehicles could also be done by machine but may not be due to cost.

  • I realize that I am replying to a post that was made years ago and yet I was still directed here by google. I hope that others won’t read this statement and mechanically agree with your silly claim.