h1

Mock me not

May 20, 2008

This is a follow-up to an earlier post of mine, where I ranted about the (ab)use of mocks in unit tests. Last week, I got into a debate with one of my colleagues on the same issue. He was of the opinion that mocks were THE way to go in unit tests. You just don’t test with real stuff at unit test level. You mock out all your dependencies and only test your code.

I agree. In principle. But like all rules, there are some exceptions. Lets take an over-simplified example. Suppose we are writing the get() method in a DAO class, in test-first fashion. The test might initially look like this:

 @Test
 public void testTest() throws Exception {
  // setup
  Statement statement = EasyMock.createMock(Statement.class);
  EasyMock.expect(
    statement.executeQuery(EasyMock.isA(String.class))).
    andReturn(EasyMock.createMock(ResultSet.class));
  EasyMock.replay(statement);
  RTest test = new RTest();
  test.setStatement(statement);
  
  // act
  Object result = test.get("id");
  
  // assert
  EasyMock.verify(statement);
  assertNotNull(result);
 }

The simplest code to pass the test:

public ResultSet get(String string) {
  try {
   return statement.executeQuery("");
  } catch (SQLException e) {
   e.printStackTrace();
   return null;
  }
}

Now, we enhance the test and add more stringent assertions:

 @Test
 public void testTest() throws Exception {
  // setup
  String expectedSql = "select * from table where id='id'";
  Statement statement = EasyMock.createMock(Statement.class);
  EasyMock.expect(
    statement.executeQuery(expectedSql)).
    andReturn(EasyMock.createMock(ResultSet.class));
  EasyMock.replay(statement);
  RTest test = new RTest();
  test.setStatement(statement);
  
  // act
  Object result = test.get("id");
  
  // assert
  EasyMock.verify(statement);
  assertNotNull(result);
 }

Now, this code fails the test:

public ResultSet get(String string) {
  try {
   return statement.executeQuery(String.format(
     "select * from table where id = '%s'", string));
  } catch (SQLException e) {
   e.printStackTrace();
   return null;
  }
}

Some of the problems with this test:

  • Since the SQL query is part of the code, your test should assert on the correctness of the query itself. By typing it twice (once in the test, once in the code), how exactly do you achieve that goal?
  • Each unit test should test a single unit of code (in this case, a method). Assert on the expected result (of course!), but the test shouldn’t define the implementation. The second run of the test should not fail.
  • An integration test or acceptance test might show any possible problems with the query. But then again, those tests may not cover all boundary cases. The unit test is the closest place to the execution of that piece of code and is the right place to carry out that real test.

How would you do that? Simple. Use an in-memory database, run the CREATE script as part of the test setup (wow, this tests your create script as well – what a bonus!), populate data required for your test, execute your test method, and assert on the data. If you are testing get() functionality, assert on the returned data. If testing save() functionality, assert on the new data in the database. Your unit tests actually test your code (and SQL) without definining how the code should be implemented.

This rationale has been used (extensively) in Aloha’s unit tests. As you might expect, we use Hypersonic to unit-test the DAO classes. Similarly, we use SipUnit to test code which sends and receives real SIP. It provides several benefits to the use of mocks to test SIP:

  • SIP is an incredibly complex protocol. There is no easy way to validate that a newly created SIP message is valid by itself, or valid in the current message flow. If JAIN-SIP sends out the message with no exceptions and it is duly received by SipUnit, we can rest assured that the message is valid.
  • Mocking out SIP responses also results in unmanagable tests. Since the required data in a JAIN-SIP response object is several levels deep, constructing the mocks is quite complicated. If any code is refactored later, the changes to the tests are very time-consuming, without providing the necessary confidence that the changes to the code are accurate. (A by-product of the test defining the implementation)
  • Because SIP is truly real-time, it is essential to test various race-conditions and concurrency issues. While our robustness and performance builds go a long way to achieving this goal, our unit tests provide our first level of sanity checking. With the current unit test setup, it is much easier to write failing unit tests when fixing a concurrency issue than it would with mocks!

This is not to say that we don’t use mocks in Aloha. Infact, they are used extensively across the codebase, just not in ALL places.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: