Comparing Java Mock Frameworks – Part 2: Creating Mock Objects


Update 2010/10/19:Eventually updated the conclusion section to provide a worthwhile synthesis (hopefully).

Update 2010/10/11: Reviewed the conclusion section not to draw any conclusion just yet.

Update 2010/10/07: Corrected mistakes pointed out by Rogerio Liesenfeld. Thanks Rogerio.

This is the second instalment in the series Comparing Java Mock Frameworks. This time, I will take a look at how the frameworks instantiate test stubs and mock objects as well as their overriding capabilities. The whole series consists of the following posts:

 


At last, I managed to finalise this second instalment. I didn’t think that I would need so so long to make this first comparison. This is quite a lot of reading and trying out. At the end of the day, I hope you’ll find it useful. If there is any mistake in this post, I’m eager to know and to fix it.

 

For this post, I experimented with the frameworks to get a feel for how they addressed the following issues:

  • How can I obtain an instance of a test double?
  • Can instances be used as stubs and mock objects?
  • Are there limitations in terms of what can be test-doubled?
    • Can the framework mock interfaces, abstract classes, concrete classes, final classes?
    • Can the framework mock final and static methods?
  • Do the mock objects return default values for mocked methods?
  • Are mocked objects strict by default (explicit or implicit verification)?
  • What are the main differences between the frameworks?

Remark: I didn’t dive into the details of each framework. It is not my intent to dive into the details of what each framework provides. I hope I didn’t overlook anything worth mentioning. If so, please tell me.

Different Approaches to Mocking

The tested framework fall into two different categories, depending on the way they create the mock objects:

  • Using  proxy: Create a dynamic implementation of a class using the Java reflection API;
    • Allows redefining the public interface of a class;
  • Remapping classes: Remap the class in the class loader using the Java Instrumentation API;
    • Allows mocking objects that are created using the new operator;
    • Allows completely redefining a class;

The limitations of what the different frameworks can mock come from the approach their take.

Mockito (version 1.8.5)

Mockito does not make any difference between mock objects and test stubs. One can obtain a mock object using the Mockito.mock() method as shown below.

// Mock an interface
List testDouble = Mockito.mock(List.class);
// Mock a class
ArrayList testDouble = Mockito.mock(ArrayList.class);

It is also possible to annotate a class variable that shall be mocked. However, the annotation requires that some specific method be called to allow for the injection. The following code snippet extracted from the official documentation illustrates how to use the annotation.

   public class ArticleManagerTest extends SampleBaseTestCase {

       @Mock private ArticleCalculator calculator;
       @Mock private ArticleDatabase database;
       @Mock private UserProvider userProvider;

       private ArticleManager manager;

       @Before public void setup() {
           manager = new ArticleManager(userProvider, database, calculator);
       }
   }

   public class SampleBaseTestCase {

       @Before public void initMocks() {
           MockitoAnnotations.initMocks(this);
       }
   }

Mockito:

  • Mocks interfaces, abstract classes and concrete classes;
  • Cannot mock final classes. It will throw an exception when trying to mock such a class;
  • Does not mock final and static method. Instead, it delegates the call to the real implementation;
  • Returns default values for mocked methods (0, null, false);
  • Mocked objects are not strict by default;

EasyMock (version 3.0)

EasyMock provides three different ways to instantiate mock objects. The method used to instantiate the object will define its behaviour:

  • EasyMock.createMock(): Does not verify the order in which methods are called and throws an AssertionError for all unexpected method calls.
  • EasyMock.createStrictMock(): Verifies the order in which methods are called and throws an AssertionError for all unexpected method calls.
  • EasyMock.createNiceMock(): Does not verify the order in which methods are called and returns appropriate empty values (0,null,false) for all unexpected method calls.

Nice Mocks can be used as stubs whereas Mock and Strict Mock are pure mock objects.

For example, the EasyMock.createMock() method:

// Mock an interface
List testDouble = EasyMock.createMock(List.class);
// Mock a class
ArrayList testdouble = EasyMock.createMock(ArrayList.class)

EasyMock:

  • Mocks interfaces, abstract classes and concrete classes;
  • Cannot mock final classes. It will throw an exception when mocking such a class;
  • Does not mock final and static method. Instead, it delegates the call to the real implementation;
  • Returns default values for mocked methods depending on how the object was created (Mock, Strict Mock and NiceMock);
  • A mock object can also be converted from one type to another: default, nice or strict;
  • The strictness of the mocked objects depends on the way they were created;

Mockachino (version 0.4)

Mockachino  instantiates both stubs and mock objects in the same way. It can mock interfaces and classes using the Mockachino.mock() method:

// Mock an interface
List testDouble = Mockachino.mock(List.class);
// Mock a class
ArrayList testDouble = Mockachino.mock(ArrayList.class);

Mockachino:

  • Mocks interfaces, abstract classes and concrete classes;
  • Cannot mock final classes. It will throw an exception when trying to mock such a class;
  • Does not mock final and static method. Instead, it delegates the call to the real implementation;
  • Returns default values for mocked methods (0, null, false);
  • Mocked objects are not strict by default;

jMock (version 2.5.1)

jMock  instantiates both stubs and mock objects in the same way. It can naturally  mock interfaces using an instance of the Mockery class. The test case class must be annotated be annotated with @RunWith(JMock.class) for JUnit4.

@RunWith(JMock.class)
public class ListTest {
    Mockery context = new JUnit4Mockery();

    @Test
    public void shouldMockInterface() {
        List mockList = context.mock(List.class);
        ...
    }
}

By design, jMock does not naturally mock classes. It is however possible to achieve that using a ClassImposteriser:

@RunWith(JMock.class)
public class ListTest {
    Mockery context = new Mockery() {
        {
            setImposteriser(ClassImposteriser.INSTANCE);
        }
    };

    @Test
    public void shouldMockClass() {
        ArrayList mockList = context.mock(ArrayList.class);
        ...
    }
}

By default mocked objects are strict. It is however possible to change this behaviour by declaring expectations and using either ignoring or allowing. The two are equals and one can choose the method that best expresses the intention.

context.checking(new Expectations() {
    {
        ignoring(aMock); // The invocation is allowed any number of times but does not have to happen.
        allowing(anotherMock); // The invocation is allowed any number of times but does not have to happen.
     }
});

jMock:

  • Mocks interfaces, abstract classes and concrete classes;
  • Cannot mock final classes. It will throw an exception when trying to mock such a class;
  • Does not mock final and static method. Instead, it delegates the call to the real implementation;
  • Returns default values for mocked methods (0, empty String, false, empty array, null);
  • Mocked objects are strict by default;

PowerMock (version 1.4.5)

PowerMock is a framework that builds on either Mockito or EasyMock. It provides additional functionality to the underlying framework.

To use PowerMock with JUnit, you have to use the PowerMock test runner.

The @PrepareForTest annotation is used to tell PowerMock to prepare certain classes for testing (typically final classes, classes with final, private, static or native methods that should be mocked and also classes that should be return a mock object upon instantiation). The code snippet below illustrates how to annotate a test case to use PowerMock.

@RunWith(PowerMockRunner.class)
@PrepareForTest( { MyFinalClassToMock.class, MyClassWithStaticMethodsToMock.class })
public class MyTestCase {
...
}

Once the test case is annotated, you can use your favourite mock framework extension (EsayMock or Mockito).

EasyMock Extension

PowerMock provides EasyMock extensions in the class org.powermock.api.easymock.PowerMock.

    MyClass myClassMock = PowerMock.createMock(MyClass.class); // Mock a class or interface
    PowerMock.mockStatic(MyClassWithStatic.class); // Mock a static class

Remark: Because of some class loader issues in the EasyMock extension, I did not manage to make it work as documented. As I didn’t want to spend much time investigating the issue, I hope it will be fixed soon.

Mockito Extension

PowerMock provides a class called PowerMockito that is used for creating mock objects and initiating verification, and expectations. For example:

   PowerMockito.mockStatic(MyClassWithStatic.class); // Mock a static class

PowerMock:

  • Can mock everything (final classes,final, private, and static methods);
  • Can remove static initializers;
  • Allows mocking without dependency injection (replaces instances even when created using new in the code under test);
  • Returns default values for mocked methods (0, null, false) as do EasyMock and Mockito;
  • Mocked objects are not strict by default when using the Mockito extension and are when using the EasyMock extension;

Unitils (version 3.1)

To make your test case Unitils-enabled, you must either extend the class public class UnitilsJUnit4 , or use the JUnit test runner UnitilsJUnit4TestClassRunner. The following code snippet illustrates both ways.

public class Test_Unitils extends UnitilsJUnit4 {
...
}

@RunWith(UnitilsJUnit4TestClassRunner.class)
public class Test_Unitils {
...
}

Having made the test class Unitils-enabled, mock objects can be automatically created and injected using the Mock type. For example:

public class Test_Unitils extends UnitilsJUnit4 {
   private Mock<MyClass> mock;
}

It is also possible to obtain a mock object programmatically:

public class Test_Unitils extends UnitilsJUnit4 {
   private Mock<MyClass> mock = new MockObject<MyClass>("myClass", MyClass.class, this);
}

Unitils provides some special annotations to allow for dependency injection in a test. Those annotations apply to fields:

  • @TestedObject: indicates the object under test, whose dependencies will be injected;
  • @InjectInto: indicates that the annotated object should be injected into the tested object based on the value of the property attribute;
  • @InjectIntoByType: indicates that the annotated object should be injected into the tested object based on the type of the object;
  • @InjectIntoStatic: indicates that the annotated object should be injected into a static property of the tested object based on the value of the property attribute;
  • @InjectIntoStaticByType: indicates that the annotated object should be injected into a static property of the tested object based on the type of the object;

The following example is excerpted from the official documentation.

public class InvoiceServiceChargeInvoiceTest extends UnitilsJUnit4 {
    @TestedObject
    private InvoiceService invoiceService;

    @InjectIntoByType
    private Mock<PurchaseOrderDAO> purchaseOrderDAO;

    @InjectIntoByType
    private Mock<AccountDAO> accountDAO;

    @Test
    public void chargeInvoice() {
        // define behavior
        purchaseOrderDAO.returns(asList(new PurchaseOrder(30), new PurchaseOrder(50))).getPurchaseOrders(5);

        invoiceService.chargeInvoice(5, "12345");

        // assert behavior
        accountDAO.assertInvoked().withdraw("12345", 80);
    }
}

Unitils:

  • Mocks interfaces, abstract classes and concrete classes;
  • Cannot mock final classes. It will throw an exception when trying to mock such a class;
  • Does not mock final and static method. Instead, it delegates the call to the real implementation;
  • Returns default values for mocked methods (0, null, false);
  • Mocked objects are non-strict by default;

JMockit (version 0.999.2)

JMockit includes two mocking APIs. The first one is for “behaviour-based testing” and is named the “JMockit Expectations & Verifications” API. The second API is for “state-based testing” and is named “JMockit Annotations” API.

JMockit Expectations & Verification API (for behaviour-based verifications)

Mock objects are essentially defined using annotations. By annotating a variable  with @Mocked, JMockit replaces any instantiation of the class in the test fixture with a mocked instance. JMockit also provides the @NonStrict annotation that leads to non strict mock objects, meaning that they will not fail when non pre-recorded methods are called on the object.

Any object annotated with @Mocked is non-strict as long as no expectation “strict” is defined on it, otherwise, it becomes strict.

@Mocked
MyInterface interfaceTestDouble; // This is a non-strict mock object but it will become strict if no strict expectations are defined on it
@Mocked
MyClass classTestDouble; // This is a non-strict mock object but it will become strict if no strict expectations are defined on it
@NonStrict
MyFinalClass finalClassTestDouble; // This is a non-strict mock object

@Test
public void someTest() {
    // All objects are not null as they were injected by JMockit
    interfaceTestDouble.someMethod();
    classTestDouble.someMethod();
    finalClassTestDouble.someMethod();
}

Because JMockit replaces the actual implementation of the mocked class, any instance of the class will be a mock. In the following code, both objects will be mock objects even though one was instantiated from the regular class.

@Mocked
MyClass classTestDouble; // This is a mock object

@Test
public void foo() {
    MyClass myClass = new MyClass(); // This is a mock object as well!!!
}
...
public class AnotherClass{
    private MyClass myClass;
    public AnotherClass() {
        myClass = new MyClass(); // This is a mock object when the instance is created in the scope of the test.
    }
}

It is also possible to obtain a mock object for a given class by declare it as a parameter to the test method. A parameter of a test method is considered, by default, to be a “mock parameter”; it can optionally be annotated with @Mocked and/or @NonStrict.

@Test
public void shouldCreateMockedParameter(AnotherClass anotherClassMock){
    anotherClassMock.doSomething();
}

JMockit:

  • Can mock everything (final classes,final, private, and static methods);
  • Can remove static initializers;
  • Allows mocking without dependency injection (replaces instances even when created using new in the code under test);
  • Returns default values for non-void methods as 0 for primitive integers, empty collections and arrays, and so on;
  • Mock objects defined with @Mocked are non-strict by default. Defining expectations on them makes them strict;

JMockit Annotations API (for stated-based verifications)

Mock objects are used in behaviour-based verifications. Nevertheless, JMockit provides an API that allows for state-based verifications. I present this API for illustrative purposes, because JMockit is the only framework that provides a dedicated API to perform this kind of validations.

The @UsingMocksAndStubs annotation specifies the classes that must be mocked or stubbed out in the test fixture. Using this annotation, classes passed on as parameters to the annotation will be stubs. This annotation allows using a provided implementation of the stub or mock object to use. The list of classes provided to the annotation contains either production classes or mock classes that are annotated with @MockClass.

The following code snippet is excerpted from the official documentation.

@UsingMocksAndStubs({MockDatabase.class, Email.class})
public final class MyBusinessService_AnnotationsAPI_Test
{
   @MockClass(realClass = Database.class, stubs = "<clinit>")
   public static class MockDatabase
   {
      @Mock(invocations = 1)
      public static List<EntityX> find(String ql, Object... args)
      {
         assertNotNull(ql);
         assertTrue(args.length > 0);
         return Arrays.asList(new EntityX(1, "AX5", "someone@somewhere.com"));
      }

      @Mock(maxInvocations = 1)
      public static void persist(Object o) { assertNotNull(o); }
     }
...
}

In the above example, the static initializer of the Database class is also stubbed out, by specifying the special method name “<clinit>”.

There is one more way to mock  classes with JMockit. It consists in defining what is called a mock class, which is a class with methods annotated with @Mock.  The mock methods of the mock class with substitute for the actual implementations in the target class to mock . Here is an example from the official documentation.
This code snippet defines a mock class, with methods that will substitute for real methods of the target class.

   public static class MockLoginContext
   {
      MockLoginContext() {}

      @Mock
      public MockLoginContext(String name, CallbackHandler callbackHandler)
      {
         assertEquals("test", name);
         assertNotNull(callbackHandler);
      }

      @Mock
      public void login() {}

      @Mock
      public Subject getSubject() { return null; }
   }

This code snippet declares the test. The method substitution is performed when calling the Mockit.setUpMock method with the target class and an instance of the mock class.

   @Test
   public void setUpMocksForGivenRealClassWithGivenMockInstance() throws Exception
   {
      Mockit.setUpMock(LoginContext.class, new MockLoginContext());

      // Inside an application class which creates a suitable CallbackHandler:
      new LoginContext("test", callbackHandler).login();

      ...
   }

Current observations

So far, I can sum up the main characteristics of the frameworks in the following categories:

  • Type: Proxy-based or class remap, which determines the extent to which behaviour can be changed in the created objects;
  • Stubbing: Determines whether the framework allows stubbing, with or without extra configuration steps;
  • Default return values: Not all frameworks behave the same with default values for non-void methods;
  • Instances are strict/non-strict by default: Some frameworks make objects strict by default while others don’t and some others allow specifying the strictness. That can have some importance depending on your testing approach;
  • Programmatic creation: The framework provides static methods to create objects;
  • Declarative creation: The framework provides annotations or wrapper types to identify the objects to mock or stub out;
  • Injects dependencies: The framework injects the dependencies of the class under test that are identified as either mock objects or stubs;
  • Replaces instances created using new: The framework does not need that dependencies be injected into the class under test. It can replace instances created using the new keyword;

All the evaluated frameworks are capable of creating both mock objects and stubs. The differences lies in whether or not the frameworks objects are strict or not by default.

The table below summarises the collected information:

Framework Type Stubbing Default return values Instances are strict/non-strict by default Programmatic creation Declarative creation Injects dependencies Replaces instances created using new
Mockito Proxy-based Yes 0, null, false Non-strict Yes Yes No No
EasyMock Proxy-based Yes Strict instances throw exceptions. Non-strict return 0, null, false Depends on the ways the object was created. Type can be changed at runtime Yes No No No
Mockachino Proxy-based Yes 0, null, false Non-strict Yes No No No
jMock Proxy-based Yes 0, empty String, false, empty array, null Strict Yes No No No
PowerMock Class remap Yes 0, null, false Strict with EasyMock and non-strict with Mockito Yes No No Yes
Unitils Proxy-based Yes 0, null, false Non-strict No Yes Yes No
JMockit Class remap Yes 0, empty collections, empty arrays, Non-strict by default No Yes Yes, replaces instances of @Mocked classes Yes

The two different approaches to mocking allow for different kinds of usage. On the one hand, proxy-based framework do not allow mocking certain elements that are present in legacy code. They are of less help if the code under test does not allow for dependency injection for example. On the other hand, frameworks that are based on class remapping allow testing any kind of code. That comes in particularly handy for legacy code.

In what I have seen so far, I like the possibility to annotate the objects that will be mocked and have them injected. It can help write clean test code.

Conclusions

Since I wrote the first installment, I had time to think again about what I want to achieve in this series. Comments on the post also made me realise that I do not want to choose any framework. I just want to put together information that will help me understand each of them. This information will help me, and any one who might find it helpful, choose the most appropriate tool for the job at hand. I believe it is a question of taste. If it weren’t, all frameworks would look the same :-).

On the other hand, I believe that some other points such as the quality of the documentation as well as the easiness to get started with build tools such as Maven for example might be valid criteria. I shall consider those to help choose a framework.

To conclude, I can’ t really draw any conclusion just yet. We will see in the next posts how to actually use the framework to test some code.

Follow

Get every new post delivered to your Inbox.

Join 163 other followers

%d bloggers like this: