Monday 2 February 2015

JMH: How to setup and run a JMH benchmark

Health Warning!
This post describes how to setup and run a simple JMH benchmark.  Micro benchmarks are notoriously difficult to get right and even when you do get them right (by using tools such as JMH) they can still be misleading. Just because your code runs in a certain way in an extremely isolated artificial situation does not mean it will run in the same way inside your production code. To name but a few issues, in a real program the CPU caches will be subject to pressures from other parts of your code, any object creation will have a downstream effect on GC and the JIT may have inlined and compiled code from other parts of your code that conflict with the code you have benchmarked. Nevertheless micro benchmarks do have their place and if you are going to do them you might as well do them properly with JMH.

In a recent post I was asked to execute my tests as a JMH performance benchmark.

JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM. See full documentation here.

Amongst other things JMH is great, because it takes care of warm up iterations, forking JVM processes so that benchmarks don't interfere with each other, collating results and presenting then in a uniform manner.  And there's much much more.

I'd heard a lot about JMH and seen many JMH results but never actually run one myself. It was surprisingly easy! This is how I did it.

There are two way to run a benchmark:

  1. Add the JMH maven dependencies to your pom file and then add a main method to your code using the Runner object.  This is useful if you want to run in your IDE.  There is some discussion about whether running inside your IDE makes a difference to the results see here for more details.  The reported difference is 2.2%.
  2. The recommended way is to generate a pom file and use that to create a jar. The mvn install uses the shade plugin to create a jar file so that you don't have create a main method.

Method 1 - For running in your IDE

Add these dependencies to your Maven pom.xml file:


Then decide which methods you want benchmarked and add the annotation @Benchmark to them. If you need any initialisation code add it in a method which should be marked @Setup.  

The easiest way to run the benchmark is by adding by adding this implementation into your main method. (See here for other ways to run your tests).

public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()

        new Runner(opt).run();

Alternatively, even more simply use this code:

public static void main(String[] args) throws Exception {

Then just run as you would any normal program and you will get all the JMH goodness!

Method 2 - The recommended way

Generate a pom file using this mvn command.
$ mvn archetype:generate \
          -DinteractiveMode=false \
          -DarchetypeGroupId=org.openjdk.jmh \
          -DarchetypeArtifactId=jmh-java-benchmark-archetype \
          -DgroupId=org.sample \
          -DartifactId=test \
This will create a project called test with an empty benchmark in it called MyBenchmark.
To build the project just use mvn clean install. This will build a jar called benchmark.jar.  It is the benchmark.jar that should be run to run the benchmark not any other jars produced along the way that will be in your target folder.

To run use the command java -jar benchmark.jar - that's it.

As an example to see the format of a JMH benchmark, this is what my results looked like:

Benchmark                                         Mode  Cnt     Score    Error  Units
CompTestBenchmark.bmCustomComparator             thrpt   20  2598.617 ± 67.634  ops/s
CompTestBenchmark.bmJDKComparator                thrpt   20   751.110 ± 14.835  ops/s
CompTestBenchmark.bmNoVTLComparator              thrpt   20  1348.750 ± 30.382  ops/s
CompTestBenchmark.bmNoVTLOrAutoBoxingComparator  thrpt   20  2202.995 ± 43.673  ops/s

There are an enormous number of bells and whistles to fine tune your benchmarks which I'm not going into here but hopefully this will get you up and running.

For a full code listing of my test see here.

To see all the options you have available use the command:

java - jar benchmark.jar -h

One of the most useful options is to run with the benchmarks with a profiler.  To list the available profilers on your system (for example perf is only available on Unix) use the command.

java -jar benchmark.jar -lprof

To run the test with a simple stack profiler (available on all systems) use

java -jar benchmark.jar -prof stack

There are loads of options including annotations to switch off inlining, to vary iterations and many more.  I encourage you to have a look at them - here's some documentation to start with.