Mocking Mocking and Testing Outcomes

Chad Myers 25 minutes later:

• • You should really suffix these DI/mock posts with “in Java”

  • Because your arguments stand… in Java. In C# and Ruby and JavaScript, and just about every other modern language, your arguments do not stand.

Kay Johansen about 1 hour later:

• • I too have seen overuse of mocking frameworks in tests, and agree that it’s a problem. However, your examples showed only one side of the story. From the name of your tests, e.g. credentialsShouldBeNonNullIfServiceNamePresent(), you indicate that you’re interested in the outcome. You did not give an example of a case when you really do want to test the interactions between objects, instead of outcomes.

  • For example, if I want to test that an object properly notifies its observer, I use Mockito thus:

  • @Test public void notifiesOutputValueChangedWhenAnInputValueChanges() { component.notifyValueChanged(); verify(connectTo).notifyValueChanged(); }

  • I find the readability vastly improved by using a framework such as Mockito – and this is not a complex example with many objects.

Kay Johansen about 1 hour later:

• Sorry for the formatting problem in my previous comment.

• •  @Test   public void notifiesOutputValueChangedWhenAnInputValueChanges() {     component.notifyValueChanged();     verify(connectTo).notifyValueChanged();   }

Bill Sorensen about 2 hours later:

• • I also use the excellent Moq framework in C#, and find it extremely valuable. I’ve seen (and written) fragile tests using mocks. I’ve come away with different lessons, though:

  • 1. Avoid using behavior verification. Whenever possible create simple dummies/fakes/stubs instead and use state verification on the class under test. Mocking frameworks make it easier to write these test doubles.

  • 2. Use TDD. I get into trouble when I write my tests after I write my production code.

  • 3. Refactor test classes to remove duplication. Extract a method to create a test double and give it a descriptive name.

  • 4. As you say, don’t feel obligated to use the mocking framework all the time. If a hand-rolled stub makes more sense, use that. Just factor in the maintenance on that class when the interface changes.

Chris Brandsma about 2 hours later:

• • C#/Rhino Mocks/ReSharper user here…

  • I will agree that for simple classes creating a quick stub is not too much trouble. But for most of those cases you can also create a dynamic mock without too much ceremony as well. I often have to do this for .net Compact Framework code, for which there is no mocking framework.

  • The issue, as I see it, is when you have multiple test scenarios to test through. The amount of setup required in your mocks/stub/fake can go up quickly. Worse, you might end up with multiple test fakes for single test cases. I have folders that contain nothing but stub classes for the same dependency—not good.

  • That second part becomes a real maintenance issue as you continue development. Now, as you add properties, methods, and event to a given dependency, those items have to be propagated to your test classes.

  • Next you have a business requirement change (like those never happen) and now you have to go through and change behaviors in all those places.

  • Because of those problem areas, I find that it is ultimately simpler to just use a mocking framework. I don’t want to be afraid to change my code because of all the work involved to update the tests.

  • Keep up the good work.

Jan-Kees van Andel about 2 hours later:

• • Funny you’re writing this article now. Last week I had a discussion with a colleague about two things you talk about in this article.

  • First, we were arguing the use of mocking frameworks vs. hand writing mock objects. My point was that you should only use a mocking framework when it provides significant advantages over plain Java coding. My example was a complex class with lots of methods on which you would only need specialized behavior for one method. The proxy approach of i.e. EasyMock then has a significant advantage over handwriting a mock object.

  • Another advantage of hand writing mocks, I think, is that is enables better encapsulation of test data. In most systems you can cover 90% of the situations using less than 10-15 test profiles. These profiles are known by the test department, are realistic and self documenting (i.e. a banking customer with one payment account and one savings account). When put into an appropriately named class, it becomes very simple for other people (not only developers, but also project managers or business people) to visualize what’s going on. With mocking frameworks, you’ll have the risk of scattering all this useful information across several test cases, losing this benefit.

  • Second, he thought the point of a unit test was to test class internals, instead of the interface. Although I’ll have to agree on some degree (you must know the internals of a class when setting up your mocks), I argue this is a good thing and I think it leads to very fragile and maybe even useless tests.

  • So, I totally agree with your article and this gives me a bit more ammunition for my next discussion (which I’ll start myself this Monday ;-)).

Pablo Fernandez about 3 hours later:

• • Nice post, really.

  • Just a mention, the first mockito code could be refactored to use

  • any(OId.class)

  • to avoid the casts. :)

Morten about 3 hours later:

• • Thank you.

  • I recentlly had an eye opener at work, where I was shown how our extensive use of Rhino Mock’s garbled the readability of our testing.

  • After rewriting the test without Rhino mock’s it became apparent to me, how our learned behaviour to use mocks as much as we did affected the quality of our testing code.

  • I’m now looking at mocking suspiciously.

Tim Gifford about 4 hours later:

• • Rolling my own mocks always made developers look at me weird. Their thought was that only developers new to TDD manually mocked.

  • I don’t always hand code them, but when I think it will lead to clearer understanding and easier maintenance it is an easy decision.

Colin Jack about 4 hours later:

• • I used to agree but these days I have a very different view.

  • I think mocking is best used to driven design (see Mock Roles, Not Objects http://joe.truemesh.com/MockRoles.pdf).

  • Those tests do end up coupled to the design, they describe the interaction, so are relatively fragile. However I don’t mind this as they are just one type of test I write, I also have unit tests (including state based) and acceptance tests so if the design changes I can just delete the old interaction tests. Oh and I only write these tests when I find them useful, so far mainly for controllers/services and not so much for business/domain logic.

  • In addition I find manual stubs useful but only when the stub is being used quite widely, basically where I’m using stubs for conveniance. For example mocking out our repositories to keep our tests fast.

Esko Luontola about 5 hours later:

• • • There may be a dance of several method calls between a few different objects; but the test is oblivious since it only checks the answer. Therefore the tests are not strongly coupled to the solution and are not fragile.

  • +1

  • It’s best to keep the tests as decoupled from the implementation details as possible. That makes it easier to refactor the code – then the tests don’t need to change when the implementation is changed.

Derek Greer about 6 hours later:

• • There are several flaws with the reasoning provided in this argument. First, the initial example you set forth seems to be more a reflection on the design of the system under test and the testing approach in general than their use of the mocking framework. Your assertion that the team’s use of a mocking framework somehow led to this design is absurd. Were they to have restricted themselves to hand-rolled mocks and stubs, the resulting design and test would have been just as bad if not worse. This example serves no purpose but to prejudice the reader in preparation for your following arguments.

  • Concerning your FitNesse test example, quite frankly I had more trouble determining the intent of your test due to your testing style than your use of the mocking framework. For instance, from looking at the test, I was confused what was actually meant by no service name being present. It wasn’t until I looked at your positive test and saw that what you were actually testing for was the absence of something within the properties that I understood what the test was actually verifying. Of course, when using a bag of values like this you can’t very well explicitly declare that the service name is null as you might with setting a property or method call, but this can be alleviated with a better testing style.

  • Consider the following example using Machine.Specifications with Moq:

  • http://pastie.org/791697

  • The test reads:

  • When credentials are requested with no service name property set

  • ...Because the GetServiceCredentials() method was called

  • .....The SUT should fail to retrieve a service name

  • .....The SUT should not call the GSManager’s CreateName method

  • .....The SUT should not return credentials

  • I added the explicit check that the service name wasn’t retrieved for the benefit of expressing clear intent, but the name of the class actually goes a long way in communicating to the user that the key qualifier is the absence of the service name key. You set forth your final negative test as an example of how much cleaner your example was than your previous example using the mocking framework, but I challenge you to argue that this:

  • @Test public void credentialsShouldBeNullIfNoServiceNameWithHandRolledMocks() throws Exception {   NegotiateAuthenticator authenticator = new NegotiateAuthenticator(managerSpy, properties);   assertNull(authenticator.getServerCredentials());   assertFalse(managerSpy.createNameWasCalled); }

  • Is clearer than this:

  • It should_not_call_create_name = () => _mockManager.Verify(m => m.CreateName(), Times.Never()); It should_not_return_credentials = () => _credentials.ShouldBeNull();

  • Also, because the context of your test is now smeared across several classes, it’s ultimately harder to understand. Ultimately, I think you’ve failed to demonstrate how hand-rolling your mocks is a cleaner approach.

Harley Pebley about 6 hours later:

• • Thanks for this article. Some of your tweets earlier this week on this topic intrigued me. I hoped you were going to write more about it.

  • On the topics, I agree and disagree.

  • Regarding testing implementation (interaction) versus interface (results), I agree. Tests are much less fragile when testing the outward facing results. Our team has had some heated discussions about the relevance of directly testing private and protected methods. Fortunately, over time I think we’re coming to a good balance.

  • However, regarding limiting the use of mocks, I disagree. A number of years ago I wrote software in an environment which did not have any mocking framework. (Delphi 5/Win32) All mock objects were hand-rolled; it was pretty painful with things that looked like your MockGSSManager.

  • (As an aside, I find it interesting you said: “And, of course, I can put this code somewhere where nobody had to look at it unless they want to.” Reminds me of your tweet a couple days ago: “The desire to hide code is really a still small voice telling you to decouple.”)

  • Now, with Rhinomocks in C#, testing is almost fun; it’s certainly more satisfying with less “busy” code. And, the mock’s code is right there close to, if not in, the test, rather than spread in another class.

  • Anyway, thanks for provoking thought.

Neil Mosafi about 6 hours later:

• • Doesn’t this indicate a lack of expressiveness of the mocking framework you are using? I find Rhino Mocks fairly readible. That final verify you just did there could have been written using the following syntax:

  • manager.AssertWasNotCalled(m => m.CreateName(null,null,null),                             c => c.IgnoreArguments());

  • It’s fairly expressive. The use of Expression Trees in C# definitely help here.

Aaron Day about 10 hours later:

• • By using a mock framework you are able to test exactly the proper functioning of a method rather than any dependencies of the method. Certainly you can get the same effect by hand-rolling your test doubles, but it seems like extra work.

  • I still do not see the harm in the use of the mock framework. Hand rolling a class adds code which has to be maintained, and as much as I like writing code I always want to maintain less of it.

  • The idea that using mocks enable you to write code that have a high degree of coupling is a strawman. Either mocks or hand rolled class allow you to do it. The warning sign may be more acute with hand rolling, but that is the case when hand crafting anything.

  • Perhaps mocks make it easier to write code with a higher degree of coupling, but I think they also make testing easier. A good mock library is used consistently throughout the testing code base, where a hand rolled implementation is likely to be more ad-hoc and customized to each test. Consistency improves readability, and in the end isn’t that one of the goals of clean code?

Philip Schwarz about 10 hours later:

• • From xUnit Test Patterns – Refactoring Test Code (2007):

  • The jury is still out on whether Behaviour Verification is a better approach than State Verification. In most cases, State Verification is clearly necessary; in some cases, Behaviour Verification is clearly necessary. What has yet to be determined is whether Behaviour Verification should be used in all cases or whether we should use State Verification most of the time and resort to Behaviour Verification only when State Verification falls short of full test coverage.

Philip Schwarz about 10 hours later:

• • From xUnit Test Patterns – Refactoring Test Code (2007):

  • State Verification comes naturally when we are building the software inside-out. That is, we build the innermost objects first and then build the next layer of objects on top of them…A common application of Behaviour Verification is when we are writing our code in an outside-in manner. This approach, which is often called need-driven development, involves writing the client code before we write [its dependencies]...

    • The main objection to this approach is that we need to use a lot of TestDoubles to write tests. That could result in Fragile Tests, because each test knows so much about how the software under test is implemented.

Philip Schwarz about 11 hours later:

• • Hello Uncle Bob. Interesting post, as usual.

• You said:

  • Lately I have seen several books and articles that present TDD through the lens of a mocking framework. If you were a newbie to TDD, these writings might give you the idea that TDD was defined by the use of mocking tools, rather than by the disciplines of TDD.

  • Is Growing OO Software, Guided by Tests one of these books?

• Here is an extract:

  • We appear to have painted ourselves into a corner. We are insisting on focused objects that send commands to each other and don’t expose any way to query their state, so it looks like we have nothing available to assert in a unit test. One option is toreplace the target object’s neighbours in a test with substitutes, or mock objects. We can specify how we expect the target object to communicate with its mock neighbours for a triggering event; we call these specifications expectations...

    • With this infrastructure in place, we can change the way we approach TDD. ... We can use the test to help us tease out the supporting roles our object needs, defined as Java interfaces, and fill in the real implementations as we develop the rest of the system. We call this interface discovery.

Fernando Zamora about 14 hours later:

• • In all we do our job is to reduce complexity. I think that when things become overly complex that in it self should be a design smell. This may mean that if you are creating an entire class to do something that could have been done with one line of mocking code, then you are adding unnecessary complexity. If on the other hand your mocking code looks like Uncle Bob’s mock example… well that smell may really be just pointing to a bigger design issues.

  • @fernandozamoraj

Gil Zilberfeld about 15 hours later:

• • Uncle Bob,

  • Charged topic. I/m not going to talk about test-after, in this case, you’re bound to run into complex scenarios where it makes much more sense (time saving wise) to use a framework.

  • I usually go with the Act-Assert-Arrange method. Set up the fakes after they reveal themselves. I can use both hand-rolled or Isolator, and that’s if I need it – the test tells me if I do. But eventually, I choose the framework. It’s easier, and readability is better.

  • By the way, hiding the manual mock code somewhere, doesn’t make it go away. When something breaks (and it eventually will, you’ll need to go in and fix it), and the readability is hampered in this case – it doesn’t let you fix the code quickly.

  • @gil_zilberfeld

J. B. Rainsberger about 16 hours later:

• • This bit of bad logic disappoints me, Bob.

  • “That statement does not require my poor readers to understand anything about Mockito. Of course it does require me to hand-roll a manager mock.”

  • No. If I wanted to write assertTrue(manager.createNameWasNotCalled());, then I would delegate that method to the corresponding assert never called in my mock framework of choice.

  • I understand that you favor rolling your own over using the frameworks. I agree that people shouldn’t use a tool without understanding its purpose, and that people do very strange things with mock frameworks. Let me ask you: since people do very strange things with tests, such as write them “just for coverage” without assertions, why not encourage people to roll their own test framework and avoid JUnit?

Rob Bowley about 17 hours later:

• • Respectfully I think you’re argument is somewhat flawed, as many others here have pointed out.

  • You’re clearly from the classicist school as Fowler put it in his infamous “Mocks aren’t Stubs” article, which is fine. It’s a preference of style, but little more than that as both approaches can end up with desired or undesired results if followed in a dogmatic and/or naive manner.

  • There’s nothing wrong with advocating a style per se, but that’s not what you’re doing here. You’re writing off the “mockist” school (a la Freeman & Pryce) altogether.

  • In your previous article on IoC containers, the framework argument you use again here was justified, but mocking is /not/ a framework as you misleadingly imply. Like TDD, Mocking is an approach to writing software

  • This article is about as relevant as if you’d written an article with lots of examples of bad unit tests and blamed it all on using JUnit instead of writing your own test framework.

Ittay Ophir about 22 hours later:

• • Good article, and nice set of enlightening comments.

  • I do not have a lot to add here, but I do have to say something, because, well, the code in the first example is mine.

  • The truth is this:

  • Using mock in this case have NOTHING to do with misunderstanding of TDD.

  • I wanted to describe a 7 years old code, that was never designed properly, and had 0 tests attached to it. I did it using unit tests, which looked like a good way to explore.

  • I started out with “hand-rolled-mocks” as you call it (I just refer to it as “Doubles”), and it took me FOREVER. The problem was the highly coupled code with the bad responsibility separation, that made it almost impossible to make a single test pass.

  • Then I discovered RhinoMocks (and later, Moq), and writing test-after went down from forever to forever-minus-one.

  • Today we use the Moq widely, mostly to make it easy to keep the separation between different layers. The AAA approach make it easy to minimize behavioral coupling, and hand-rolled-mocks will usually hide the mocking framework, to leave the interface exposed and testable.

  • For now, I see 2 main advantages in using Doubles over Mocks:

  • 1) Every time the interface changes, the double will break. It will emphasis what package depends on what service.

  • 2) It keeps all mocked behavior in one place (at least per package). This might help protect the tests itself from inconsistent behavior assertion between test cases.

  • Thanks bob, I’m going back to build my time-machine…

Sebastian Kübeck about 24 hours later:

• • I have to agree with J. B. Rainsberger. The question whether to use a mock library or not is not the problem in the second example. It’s the test itself.

Jeffrey Palermo 1 day later:

• • I have only been doing TDD for 5 years,but that is enough time to have Already drastically overused mocks and RhinoMocks. I have learned this lesson the hard way. I am glad you wrote about it an attempt to help others avoid tar pit test suites. I haven’t used Fitnesse since 2006 but am looking at it again now.

Steve Py 1 day later:

• • I agree with your view on the first example where the use of a mocking framework can lead down bumpy roads when used incorrectly. However, that’s no different than any other tool.

  • Manually rolling a stub is one thing. Manually rolling a mock is another. Even so, I personally use Moq to provide both stubs and mocks without an ounce of trouble due to overly tight coupling. I don’t see how the use of a hand-rolled stub would be any easier to understand in the context of a test than a simple, to the point Moq-provided stub.

John Sonmez 1 day later:

• • Hmm, I’m not entirely convinced by your post here, but it has swung me more on the hand coded side than I was before. The biggest problem that I have with hand rolling the mocks is that you may need different behavior in different tests, then you would have to have multiple hand rolled mocks that do different things, which can be confusing and create a maintenance problem.

  • On the other hand, you do present a really good point. The test where you had hand-rolled the mock was much easier to read and understand.

micah@8thlight.com 1 day later:

• • Thanks for writing this. I’ve been meaning to write a similar blog for over a year… now I can just point to yours.

  • You advise to “use a framework only when that framework will give you a significant advantage.” You also provide the statistic that only 11/277 test files in FitNesse use Moquito. That’s not even 4%. 4% is hardly a significant advantage. In my opinion, the FitNesse source code would better off without the use of Moquito all together.

  • Viva Hand-Rolled Mocks!

Rogério 3 days later:

• The following version of the test can be written with JMockit Annotations, a “state-oriented” mocking API:

• • class GSSManagerSpy extends MockUp<GSSManager> {    boolean createNameWasCalled;     @Mock // could be @Mock(invocations = 0) instead    GSSName createName(String s, Oid oid) {       createNameWasCalled = true;       return null;    } }  @Test public void credentialsShouldBeNullIfNoServiceName(GSSManager dummyManager) throws Exception {    GSSManagerSpy managerSpy = new GSSManagerSpy();     NegotiateAuthenticator authenticator = new NegotiateAuthenticator(dummyManager, properties);    assertNull(authenticator.getServerCredentials());    assertFalse(managerSpy.createNameWasCalled); }

Rogério 3 days later:

• ... but it could also be written like this:

• • @Test public void credentialsShouldBeNullIfNoServiceName(final GSSManager manager) throws Exception {    NegotiateAuthenticator authenticator = new NegotiateAuthenticator(manager, properties);    assertNull(authenticator.getServerCredentials());     new Verifications() {{       manager.createName(anyString, any); times = 0;    }}; }

Eric 4 days later:

• • Hi,

  • There are a few points on this article that I wish to comment:

  • 1. Hand-rolled mocks

    • And, of course, I can put this code somewhere where nobody had to look at it unless they want to.

  • I’m not sure this is so harmless as it seems to be. The best code is still the code that’s not written. Whether tools can help you or not, you’ll still have to manage those “dummy” classes, know about them, version-control them, refactor them,...

  • 2. Design

  • I agree with you that too much stubbing/mocking in your tests may be a sign that there are some design issues in your code, but I’d like to know:

    • Lately I have seen several books and articles that present TDD through the lens of a mocking framework.

  • As someone else asked, are you’re referring to “Growing OO Software, Guided by Tests” as one of those books?

  • The view that the authors of that book are advocating is really about the tell, don’t ask principle. In that view mocks are considered as a discovery/design technique which is, IMO, closer to the original OO approach than any of the code I see nowadays.

  • For example, the FitNesse test example you’re showing is all about testing the getters of an Authenticator. That doesn’t look very OO, in a behavioral sense and I would like to see how the Authenticator class could be reworked in that perspective.

  • 3. Testing

  • I’m wondering why you’re not only testing that the returned credentials are not null in the “credentialsShouldBeNonNullIfServiceNamePresent” test, as the name implies?

  • Also, why do you need to verify that “verify(manager).createCredential” was called? If you really get the credentials you’re expecting from the Authenticator, why would you need to check where they come from? Isn’t it one more way to couple your test to the implementation?

Donnie 8 days later:

• • It is a very debatable topic. But as you are under a necessity to memorize about testing or purchase a magnificent paper about it, you can click online essay writers

Manne Fagerlind 15 days later:

• • I agree with most of the things that Uncle Bob says, but creating a test double manually for the sake of readability is imho overkill. It may also confuse the reader, who will have to look at the test double class to see that createNameWasNotCalled() is not a member of the GSSManager class.

  • Instead I’d suggest extracting the verification to its own method, like so:

  • private void verifyCreateNameIsNotCalled() { verify(manager, never()).createName( anyString(), (Oid) anyObject(), (Oid) anyObject()); }

  • As previously said, Mockito usually makes for more readable verification code, but this is another to make the test work as documentation.

Tomas about 1 month later:

• • In the final two sentences in this blog post it is claimed that you keep coupling between code and tests low by testing outcomes instead of mechanism.

  • I think object-oriented development is about allocating behavior to objects. The interactions between those objects should often be a part of the method contract for the method using the collaborating objects to perform its task. In other words, the interactions should not always be considered as implementation details nor “high” coupling just because there is coupling. Certain coupling is indeed desired, while “high” coupling is not desired, but it is not obvious where the limit is between appropriate coupling and too high coupling.

  • For example, if a method of an observable class will notify an observer during certain conditions when the method is invoked, then it should be communicated in the javadoc (or something similar for other languages than java) as a significant and expected consequence of invoking the tested method. Consider the FitNesse Fixture method “protected void interpretTables(Parse tables) {” which contains the code line “listener.tableFinished(tables);” (http://github.com/unclebob/fitnesse/blob/20100103/src/fit/Fixture.java )

  • How could such a method have been developed with true TDD, with one of the rules being: “Developers are not allowed to write production code until they have written a failing unit test.” ( quoted fromhttp://www.objectmentor.com/omSolutions/agile_xp_differences.html ) Before being allowed to write the code line “listener.tableFinished(tables);” there should have been a failing (red) test, which should then have been fixed (green) by adding that code line, i.e. before the above mentioned production code line was added, then the TDD code should have been written to verify (but initially fail) that the method becomes invoked.

  • In other words, there should have been some truly TDD-written test code such as “verify(fixtureListenerMock).tableFinished(tables);” which should have failed before actually being allowed to add the invocation “listener.tableFinished(tables);” to make the test pass.