Stop mocking me already

February 12, 2008

We all use mock objects for testing. Since the advent of IOC and DI, it is an extremely common pattern to mock or stub out dependencies for a class, inject them in, and then test the class without worrying about access to a database or an http server from a unit test.

This is all very good in theory but how well does it work in practice? I regularly use both mocks and stubs and it is important to realize which one is needed for a particular unit test or a test class. It is an art, but with experience, you tend to make the right choice more often than not.

So, where is this rant coming from? I recently moved over to a codebase which makes extensive use of both mocks and stubs, but especially mocks. When using mocks, you need to specify expected behaviour of each mock. When abused, your unit tests tend to define the implementation of your methods. I HATE THAT. Unit tests are meant to test your implementation of a tiny piece of functionality, not dictate the implementation. When you add two lines of code to your method, you shouldn’t have to change a bunch of unit tests which now fail because you called an unexpected method on your mock object. With a stub, to a large extent, this problem is muted, but can still surface from time to time.

For the past year, I worked on building a SIP server. As a team, we decided we should test real SIP in our unit tests. Acceptance tests were at a much higher level of abstraction and weren’t ideal for testing low level SIP messages. If we mocked out the SIP layer, we would a) spend 50% of our time mocking out complex JAIN-SIP objects to test our code, b) find our acceptance tests were failing because of incorrect behaviour at SIP level and spend another 20-30% of our time redefining our mocks to keep our tests passing and c) more than likely, stop maintaining our tests because of the nightmare caused by the first two points.

Instead, we chose to use SipUnit to represent SIP endpoints that can be used in testing our SIP server. We were testing real functionality from our unit tests and though it increased build times significantly, the benefits gained by saving developer cycles and the rise in confidence that our unit tests were testing exactly what they should be testing were more than adequate compensation for the build times.

In the same way, using an in-memory Hypersonic database to test all dao code is extremely preferable to using mocks to test database access. I would much rather peek directly into a table in a database from my test (using a select) than assert with a mock or stub the exact SQL string used in an insert statement. I really don’t care if there’s more whitespace in the SQL than what I expect in my mock, and more often than not, I will copy my expected SQL from the test into the code (or vice-versa, depending on if you write test-first), thereby having the tests pass, while possibly still having errors in your code. You might of course want to stub or mock out your dao code when testing other parts of your application, or if you deem it necessary, continue using the in-memory database.

While I am at it, another point: I really enjoyed the book xUnit Test Patterns (part of the Martin Fowler series), from which I picked up several useful tips and tricks for unit testing. In it, the author goes to great pains to state that when testing concurrent code, it is essential that unit tests don’t introduce concurrency, but instead test functionality serially and deterministically, preferably using mocks and stubs. I disagree. While it is important and necessary to have the kind of tests he mentions, it is also important to unit test the concurrent code. What better place is there to test race conditions in concurrent code than at the lowest possible layer of abstraction?

This is not all to say that mocks and stubs don’t have their uses. As with anything else, don’t use them blindly. Use them with caution; use them when they are the right tools for the job.


One comment

  1. I so much appreciated your rant Rag! Myself I share your point of view. Mocks are very useful but sometimes people in sake of being able to test a class in pure unit test manner are introducing such a horrible mock-hell which doesn’t help at all. Tests which assert tedious things or completely define implementation (I do not believe in that!) and don’t tell you much how this particular component works and how to use it. Apart of actual correctness validation that’s the value of unit tests for me. I don’t care that pure unit tests are not giving me 100% code coverage. I can create couple of integration tests which will exercise that code and give me REAL state overview of the code. Keep it simple! Use common sense!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: