Pages

Sunday, April 3, 2011

Run a JUnit test repeatedly

I like JUnit. It's easy to extend, and easy to use.
After reading abyx's post on Retry Rule, I decided to write a post about running repeating tests with JUnit.


Hope you enjoy this:


Sometimes you wish to run a test many times. For example, it might have some random factor in it. To cover many cases, you'd like to run it a lot of times. What options are there?

First, and the most obvious, is a loop. Run a for loop, and see it fail:
 public class ExampleTest {  
      @Test  
      public void sometimesFail() {  
           for (int i = 0; i < 10; i++) {  
                int rand = new Random().nextInt(3);  
                if (rand % 3 == 0) {  
                     fail();  
                }  
           }  
      }  
 }  
This will generate a single test in the tree. It will fail on the first error and won't give you an assessment of how many times it took till it failed.


Another option is the Parametrized Runner. That would look like this:
 @RunWith(Parameterized.class)  
 public class ExampleTest2 {  
      @Parameters  
      public static Collection<Object[]>
                                 generateParams() {  
           List<Object[]> params =
                   new ArrayList<Object[]>();  
           for (int i = 1; i <= 10; i++) {  
                params.add(new Object[] {i});  
           }  
           return params;  
      }  
        
      public ExampleTest2(int param) {}  
        
      @Test  
      public void sometimesFail() {  
           int rand = new Random().nextInt(3);  
           if (rand % 3 == 0) {  
                fail();  
           }  
      }  
 }  
This is a nice solution, but the test is filled with unnecessary code. Also, all the tests in the class will run for each parameter. What if I only want to repeat a single method?


Here is a nice solution, in my opinion:
 @RunWith(ExtendedRunner.class)  
 public class ExampleTest3 {  
      @Test  
      @Repeat(10)  
      public void sometimesFail() {  
           int rand = new Random().nextInt(3);  
           if (rand % 3 == 0) {  
                fail();  
           }  
      }  
 }  
 Only state the method you want to repeat, and how many times to do this. The rest is done for you.


So, whats behind this code?
First, add an Annotation:
 @Retention(RetentionPolicy.RUNTIME)  
 @Target({ElementType.METHOD})  
 public @interface Repeat {  
      int value();  
 }  


Finally, lets add the Runner. This simply overrides the default JUnit runner (its a bit long, but not too much):
 public class ExtendedRunner extends BlockJUnit4ClassRunner {  
   
      public ExtendedRunner(Class<?> klass) throws InitializationError {  
           super(klass);  
      }  
   
      @Override  
      protected Description describeChild(FrameworkMethod method) {  
           if (method.getAnnotation(Repeat.class) != null &&  
                     method.getAnnotation(Ignore.class) == null) {  
                return describeRepeatTest(method);  
           }  
           return super.describeChild(method);  
      }  
   
      private Description describeRepeatTest(FrameworkMethod method) {  
           int times = method.getAnnotation(Repeat.class).value();  
   
           Description description = Description.createSuiteDescription(  
                     testName(method) + " [" + times + " times]",  
                     method.getAnnotations());  
   
           for (int i = 1; i <= times; i++) {  
                description.addChild(Description.createTestDescription(  
                          getTestClass().getJavaClass(),  
                          "[" + i + "] " + testName(method)));  
           }  
           return description;  
      }  
   
      @Override  
      protected void runChild(final FrameworkMethod method, RunNotifier notifier) {  
           Description description = describeChild(method);  
             
           if (method.getAnnotation(Repeat.class) != null &&  
                     method.getAnnotation(Ignore.class) == null) {  
                runRepeatedly(methodBlock(method), description, notifier);  
           }  
           super.runChild(method, notifier);  
      }  
   
      private void runRepeatedly(Statement statement, Description description,  
                RunNotifier notifier) {  
           for (Description desc : description.getChildren()) {  
                runLeaf(statement, desc, notifier);  
           }  
      }  
        
 }  


As always, here is the source code: http://tinyurl.com/3mu2zbc
Also, you can Search Amazon.com for JUnit books.

15 comments:

  1. Definitely seems like something that should be built-in in the JUnit framework.

    ReplyDelete
  2. This is great, too bad it is 4.9 and it it not released yet and you did not mention it in the post.

    ReplyDelete
  3. With JUnit 4.9b3, the most recent beta, it doesn't seem to work either. I get an error from FilterRequest:

    java.lang.Exception: No tests found matching Method spec4(Module.SpecUser) from org.junit.internal.requests.ClassRequest@43bda1e0
    at org.junit.internal.requests.FilterRequest.getRunner(FilterRequest.java:37)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:38)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:199)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:62)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

    ReplyDelete
  4. Can Extended Runner work with multiple runners? I am having a Suite Filter Runner. This would be additional Runner . Can I run test suites with Two runners?

    ReplyDelete
  5. Hey Rajesh. You can't run with 2 runners. They must work as wrappers. Its not too hard to do this though.

    ReplyDelete
  6. when you click on a test in junit view, it does not take you to the failed line, this is because the name of the test harms parsing.
    replace:
    description.addChild(Description.createTestDescription(
    getTestClass().getJavaClass(),
    "[" + i + "] " + testName(method)));
    with:
    description.addChild(Description.createTestDescription(
    getTestClass().getJavaClass(),
    testName(method) "[" + i + "] "));

    and it will work as expected

    ReplyDelete
  7. I have a situation where two frameworks are providing test runners, and I'd like to use both - somehow. Frankly I don't know how to approach this problem, as I am by no means an expert in JUnit.

    How do I create a "wrapper" that somehow provides me with the features of both runners?

    ReplyDelete
  8. Incidently I have to say that although the above is a cool JUnit extension by itself, the whole premise of unit tests that "sometimes fail" is flawed at best, and useless at worst - in my humble oppinion. Presumably you have no idea how often the test fails, so there is no guarantee that your CI server will ever execute the test enough times to observe the problem. You might as well not have the test at all.

    The correct approach is to identify the cause of the failure and write a test for that. A test that either works, or fails - each and every time :-)

    ReplyDelete
  9. I agree, but the code above is mostly a template or a code sample.
    The way I use it usually is for example:
    Add an annotation that gets a directory as an input (the directory includes sample inputs for test).
    In the test, list all files in the directory, and run the test each time on a different input file (each test will display the test file in the JUnit window).
    This way you can see directly which files fail the tests.

    ReplyDelete
  10. Very useful code, thanks!
    How can I change the code so that before each test the method setUp runs and the method tearDown runs after each test?

    ReplyDelete
  11. Hey Tolshi, it should already do that.

    ReplyDelete
  12. I used JUnit4.10, but get
    java.lang.NoSuchFieldError: NULL
    at testing.ExtendedRunner.(ExtendedRunner.java:16)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

    ReplyDelete
  13. There are two bugs in your runChild() code.

    1)In the case of @Repeat method it will run one extra time too many. The code will first run repeatedly and then it will run on more time runChild().

    2) In the case where @Repeat and @Ignore are both on the method, the runChild will also run, because the if statement will be false.

    I rewrote the runChild() as:

    @Override
    protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
    Description description = describeChild(method);

    if (method.getAnnotation(Repeat.class) != null) {

    if (method.getAnnotation(Ignore.class) == null) {
    runRepeatedly(methodBlock(method), description, notifier);
    }
    return;
    }

    super.runChild(method, notifier);
    }

    ReplyDelete
  14. you call runLeaf, but that method doesn't exist

    ReplyDelete