JUnit Best Practices

This page explains best practices related to JUnit. The overall best practices are to aim for unit tests that are:

  • extremely fast - You're going to have a lot of unit tests and they are going to run frequently so they have to be lightning fast. I'm not talking a few seconds fast; rather, I mean milliseconds fast!
  • extremely reliable - You want tests to indicate problems with production code. Tests should fail if and only if the production code is broken. Some have argued that tests that intermittently fail are worse than no tests at all!

Ensure that unit tests run completely in-memory

For example, do not write unit tests that make HTTP requests, access a database, or read from the filesystem. These activities are too slow and too unreliable, so they are better left to other types of tests, such as functional tests.

Tests that read from the filesystem are too complicated to be considered unit tests:

  • They need to configure the location of the current working directory and that can be complicated to get right on developer machines and build machines.
  • They often require storing the file to be read in source control and it can be complicated to keep the item in source control up-to-date.

Do not skip unit tests

There are a number of ways to skip unit tests, but you shouldn't use any of them:

  • Do not use JUnit's @Ignore annotation.
  • Do not use Maven's maven.test.skip property.
  • Do not use the Maven Surefire Plugin's skipTests property.
  • Do not use the Maven Surefire Plugin's excludes property.

Unit tests that are skipped provide no benefit, but still have to be checked out of source control and compiled. Instead of skipping unit tests, remove them from source control.

Aim for each unit test method to perform exactly one assertion

There are two reasons to aim for each unit test to have exactly one assertion:

  • When a unit test fails, it is much easier to determine what went wrong. If a failed unit test has three assertions, further effort is required to determine which one failed. If a failed unit test has only one assertion, no such effort is required.
  • When a unit test performs more than one assertion, it is not guaranteed that all of the assertions occur. In particular, if an unchecked exception occurs, the assertions after the exception do not happen; instead, JUnit marks the unit test method as having an error and proceeds to the next test method.

Therefore, strive for one assertion per test method.

Use the strongest assertions possible

Without strong assertions, unit tests provide nothing more than coverage. In other words, unit tests that lack strong assertions ensure that the production code does not throw an exception but when is the lack of an exception sufficient to determine that a method behaves correctly? Coverage is a worthy outcome of unit tests but we can do much better! In particular, we want our unit tests to ensure that our production code works correctly. Without strong assertions, our unit tests only ensure that our production code doesn't blow up in our face.

In order of decreasing strength, assertions fall into the falling categories:

  • strongest assertions - asserting on the return value of a method
  • strong assertions - verifying that vital dependent mock objects were interacted with correctly
  • weak assertions - verifying that non-vital dependent mock objects (such as a logger) were interacted with correctly
  • non-existent assertions

Do not verify how a mock logger was interacted with unless logging is critical to the correctness of the method being tested.

Consider writing unit tests for the following production method:

final class Foo
{
  // Members and other methods removed for brevity.

  public int bar(final int i)
  {
    if (shouldReturnDefaultValue(i))
    {
      logger.debug("Returning default value for: " + i);
      return DEFAULT_VALUE;
    }
    else
    {
      logger.debug("Returning calculated value for: " + i);
      return calculateValue(i);
    }
  }
}

How should one test this method? A naive approach might be to test the method as follows:

// Don't do this... the test is too brittle
@Test
public void foo_three()
{
  final Logger logger = mock(Logger.class);
  public Foo foo = new Foo(logger);
  assertEquals(Foo.DEFAULT_VALUE, foo.bar(3));
  verify(logger).debug("Returning default value for: 3");
}

However, this test is too brittle because it fails if the production code is changed in a non-functional way. For example, the test fails if the production code is changed so that log statements occur at a different level (say info instead of debug) or with a different message (say "Returning default value for 3" without the colon). You can eliminate the latter problem using a less strict assertion (verify(logger).debug(anyString())) but the test is still too brittle. You're better off not verifying any logger behaviour unless logging is critical to the correctness of a production method.

Use the most appropriate assertion methods

JUnit has many assertion methods - assertEquals, assertTrue, assertFalse, assertNull, assertNotNull, assertArrayEquals, assertSame. Know the assertions in the latest version of JUnit and use the most appropriate one to have the most readable test code. For example:

  • Use assertTrue(classUnderTest.methodUnderTest()) rather than assertEquals(true, classUnderTest.methodUnderTest()).
  • Use assertEquals(expectedReturnValue, classUnderTest.methodUnderTest()) rather than assertTrue(classUnderTest.methodUnderTest().equals(expectedReturnValue)).
  • Use assertEquals(expectedCollection, classUnderTest.getCollection()) rather than asserting on the collection's size and each of the collection's members.

Put assertion parameters in the proper order

The parameters to JUnit's assertions are:

  1. expected
  2. actual

For example, use assertEquals(expected, actual) rather than assertEquals(actual, expected). Ordering the parameters correctly ensures that JUnit's messages are accurate.

Use exact matching when using a mocking framework

For example, avoid using Mockito's methods that start with any; prefer configuring and verifying exact parameter values.

Name unit tests using a convention that includes the method and condition being tested

On my projects, we have used a naming convention of methodUnderTest_condition, where methodUnderTest is the method under test and condition is the condition being tested. For example, suppose we are writing unit tests for a method named encode that accepts an array of bytes and validates the bytes before encoding them. When combined with the best practice above (one assertion per test), we end up with test methods named as follows:

  • encode_nullBytes
  • encode_emptyBytes
  • encode_tooFewBytes
  • encode_tooManyBytes
  • encode_rightNumBytes or encode_validBytes

Ensure that test classes exist in the same Java package as the production class under test

When in the same package as the production class being tested, test classes can use package-private classes and invoke package-private methods. It has been my experience that high-quality code bases make extensive use of package-private classes and methods to hide implementation details.

Ensure that test code is separated from production code

The default folder structure in a Maven project does this - production code exists in the src/main/java folder and test code exists in the src/test/java folder. Even if you don't use Maven, put test code and production code in different folders.

Do not print anything out in unit tests

Unit tests are intended for consumption by the computer. Developers may add print statements to their private working copy for debugging purposes, but there is typically no need to submit a unit test that prints. For a specific best practice about not printing stack traces, see below.

Do not initialize in a unit test class constructor; use an @Before method instead

If a failure occurs during a test class constructor, an AssertionFailedError occurs and the stack trace is not very informative; in particular, the stack trace does not include the original error's location. On the other hand, if a failure occurs during an @Before method, all details about the failure's location are available. See this article for details.

Some teams allow test class members to be initialized in a constructor when those members are of a simple type such as String, but I personally don't ever initialize test class members. Instead, I always do one of the following:

  • Assign values to test class members in a method annotated with @Before (preferred).
  • Assign simple values to private static final members in the test class.

Note that this best practice also applies to members initialized upon definition. In other words, rather than this:

// Don't do this!
public final class TestFoo
{
  private FooDependency fooDependency = new FooDependency();

  // rest of test class removed for brevity
}

do this:

// Do this instead
public final class TestFoo
{
  private FooDependency fooDependency;

  @Before
  public void setUp()
  {
    fooDependency = new FooDependency();
  }
}

Do not use static members in a test class

Static members make unit test methods dependent. Don't use them! Instead, strive to write test methods that are completely independent.

Do not write your own catch blocks that exist only to fail a test

It is unnecessary to write your own catch blocks that exist only to fail a test because the JUnit framework takes care of the situation for you. For example, suppose you are writing unit tests for the following method:

final class Foo
{
  int foo(int i) throws IOException;
}

Here we have a method that accepts an integer and returns an integer and throws an IOException if it encounters an error. Here is the wrong way to write a unit test that confirms that the method returns three when passed seven:

// Don't do this - it's not necessary to write the try/catch!
@Test
public void foo_seven()
{
  try
  {
    assertEquals(3, new Foo().foo(7));
  }
  catch (final IOException e)
  {
    fail();
  }
}

The method under test specifies that it can throw IOException, which is a checked exception. Therefore, the unit test won't compile unless you catch the exception or declare that the test method can propagate the exception. The second alternative is preferred because it results in shorter and more focused tests:

// Do this instead
@Test
public void foo_seven() throws Exception
{
  assertEquals(3, new Foo().foo(7));
}

We declare that the test method throws Exception rather than throws IOException - see below for the reason. The JUnit framework will make sure that this test fails if any exception occurs during the invocation of the method under test - there's no need to write your own exception handling.

Do not write your own catch blocks that exist only to pass a test

It is unnecessary to write your own catch blocks that exist only to pass a test because the JUnit framework takes care of the situation for you. For example, suppose you are writing unit tests for the following method:

final class Foo
{
  int foo(int i) throws IOException;
}

Here we have a method that accepts an integer and returns an integer and throws an IOException if it encounters an error. Further suppose that we expect the method to throw an IOException if it is passed a parameter with the value nine. Here is the wrong way to write a unit test that confirms that the method behaves that way:

// Don't do this - it's not necessary to write the try/catch!
@Test
public void foo_nine()
{
  boolean wasExceptionThrown = false;
  try
  {
    new Foo().foo(9);
  }
  catch (final IOException e)
  {
    wasExceptionThrown = true;
  }
  assertTrue(wasExceptionThrown);
}

Instead of manually catching the expected exception, use the expected attribute on JUnit's @Test annotation.

// Do this instead
@Test(expected = IOException.class)
public void foo_nine() throws Exception
{
  new Foo().foo(9);
}

We declare that the test method throws Exception rather than IOException - see below for the reason. The JUnit framework will make sure that this test passes if and only if the foo method throws an IOException - there's no need to write your own exception handling.

Do not write your own catch blocks that exist only to print a stack trace

As we've already seen, it is a best practice for unit tests not to write anything. After all, unit tests are written to be consumed by the JUnit framework and JUnit doesn't care or monitor what gets printed. For example, suppose you are writing unit tests for the following method:

final class Foo
{
  int foo(int i) throws IOException;
}

Here we have a method that accepts an integer and returns an integer and throws an IOException if it encounters an error. Here is the wrong way to write a unit test that confirms that the method returns three when passed seven:

// Don't do this - it's not necessary to write the try/catch!
@Test
public void foo_seven()
{
  try
  {
    assertEquals(3, new Foo().foo(7));
  }
  catch (final IOException e)
  {
    e.printStackTrace();
  }
}

The method under test specifies that it can throw IOException, which is a checked exception. Therefore, the unit test won't compile unless you catch the exception or declare that the test method can propagate the exception. The second alternative is much preferred because it results in shorter and more focused tests:

// Do this instead
@Test
public void foo_seven() throws Exception
{
  assertEquals(3, new Foo().foo(7));
}

We declare that the test method throws Exception rather than throws IOException - see below for the reason. The JUnit framework will make sure that this test fails if any exception occurs during the invocation of the method under test - there's no need to write your own exception handling.

In test classes, do not declare that methods throw any particular type of exception

Test methods that declare that they throw one particular type of exception are brittle because they must be changed whenever the method under test changes. For example, suppose you are writing unit tests for the following method:

final class Foo
{
  int foo(int i) throws IOException;
}

Here we have a method that accepts an integer and returns an integer and throws an IOException if it encounters an error. Here is the wrong way to write a unit test that confirms that the method returns three when passed seven:

// Don't do this - the throws clause is too specific!
@Test
public void foo_seven() throws IOException
{
  assertEquals(3, new Foo().foo(7));
}

This test method is brittle. Imagine the method under test changed such that it could throw either an IOException or a GeneralSecurityException. In that case, we would have to change the test method for it to compile:

// Don't do this - the throws clause is too specific!
@Test
public void foo_seven() throws IOException, GeneralSecurityException
{
  assertEquals(3, new Foo().foo(7));
}

Instead, declare that the test method can throw any exception:

// Do this instead
@Test
public void foo_seven() throws Exception
{
  assertEquals(3, new Foo().foo(7));
}

That way, the unit test is no longer brittle - it is impervious to changes to the production method's signature.

Do not use Thread.sleep in unit tests

When a unit test uses Thread.sleep it does not reliably indicate a problem in the production code. For example, such a test can fail because it is run on a machine that is slower than usual. Aim for unit tests that fail if and only if the production code is broken.

Rather than using Thread.sleep in a unit test, refactor the production code to allow the injection of a mock object that can simulate the success or failure of the potentially long-running operation that must normally be waited for.

Do not attempt to test the timing of production methods that directly or indirectly call Thread.sleep

Explanation to come

Test classes directly; do not rely on indirect testing

Indirect testing occurs when a class is tested only because it is a dependency of another class. For example, suppose you have the following:

// In A.java
public final class A
{
  private final B b;

  // rest of class removed for brevity
}

// In B.java
final class B
{
  // class removed for brevity
}

Here, class B is a dependency of class A.

We should test class B directly even though its methods are also invoked by the tests of class A. That means that if we name our test classes using a suffix of Test, we should have two test classes - ATest and BTest.

It's a bad idea to rely on indirect testing because we lose coverage of the dependent class if we change the implementation of the main class. In the example above, suppose we changed the implementation of class A so that it no longer depended on class B. The tests in ATest no longer invoke the methods of class B so we lose code coverage.