JBoss.org Community Documentation

- Prev
- Next

## Chapter 11. Benchmarking and tweaking

## 11.1. Finding the best configuration

Drools Planner supports several optimization algorithms (as solver phases), but you're probably wondering which is the best one? Although some optimization algorithms generally perform better then others, it really depends on your problem domain. Most solver phases have settings which can be tweaked. Those settings can influence the results a lot, although most solver phases work pretty well out-of-the-box.

Luckily, Drools Planner includes a benchmarker, which allows you to play out different solver phases with different settings against each other, so you can pick the best configuration for your planning problem.

## 11.2. Building a benchmarker

### 11.2.1. Adding the extra dependency

The benchmarker is current in the drools-planner-core modules, but it requires an extra dependency on the JFreeChart library.

If you use maven, add a dependency in your `pom.xml` file:

        <dependency>

          <groupId>jfree</groupId>

          <artifactId>jfreechart</artifactId>

          <version>1.0.13</version>

        </dependency>

This is similar for gradle, ivy and buildr.

If you use ANT, you've probably already copied the required jars from the download zip's `binaries` directory.

### 11.2.2. Building a `PlannerBenchmark`

You can build a `PlannerBenchmark` instance with the `XmlPlannerBenchmarkFactory`. Configure it with a benchmark configuration xml file:

        XmlPlannerBenchmarkFactory plannerBenchmarkFactory = new XmlPlannerBenchmarkFactory();

        plannerBenchmarkFactory.configure("/org/drools/planner/examples/nqueens/benchmark/nqueensBenchmarkConfig.xml");

        PlannerBenchmark plannerBenchmark = benchmarkFactory.buildPlannerBenchmark();

        plannerBenchmark.benchmark();

A basic benchmark configuration file looks something like this:

    <?xml version="1.0" encoding="UTF-8"?>

    <plannerBenchmark>

      <benchmarkDirectory>local/data/nqueens</benchmarkDirectory>

      <warmUpSecondsSpend>30</warmUpSecondsSpend>

      <inheritedSolverBenchmark>

        <problemBenchmarks>

          <xstreamAnnotatedClass>org.drools.planner.examples.nqueens.domain.NQueens</xstreamAnnotatedClass>

          <inputSolutionFile>data/nqueens/unsolved/unsolvedNQueens32.xml</inputSolutionFile>

          <inputSolutionFile>data/nqueens/unsolved/unsolvedNQueens64.xml</inputSolutionFile>

          <problemStatisticType>BEST_SOLUTION_CHANGED</problemStatisticType>

        </problemBenchmarks>

        <solver>

          <solutionClass>org.drools.planner.examples.nqueens.domain.NQueens</solutionClass>

          <planningEntityClass>org.drools.planner.examples.nqueens.domain.Queen</planningEntityClass>

          <scoreDirectorFactory>

            <scoreDefinitionType>SIMPLE</scoreDefinitionType>

            <scoreDrl>/org/drools/planner/examples/nqueens/solver/nQueensScoreRules.drl</scoreDrl>

          </scoreDirectorFactory>

          <termination>

            <maximumSecondsSpend>20</maximumSecondsSpend>

          </termination>

          <constructionHeuristic>

            <constructionHeuristicType>FIRST_FIT_DECREASING</constructionHeuristicType>

            <constructionHeuristicPickEarlyType>FIRST_LAST_STEP_SCORE_EQUAL_OR_IMPROVING</constructionHeuristicPickEarlyType>

          </constructionHeuristic>

        </solver>

      </inheritedSolverBenchmark>

      <solverBenchmark>

        <name>Move tabu</name>

        <solver>

          <localSearch>

            <selector>

              <moveFactoryClass>org.drools.planner.examples.nqueens.solver.move.factory.RowChangeMoveFactory</moveFactoryClass>

            </selector>

            <acceptor>

              <moveTabuSize>5</moveTabuSize>

            </acceptor>

            <forager>

              <pickEarlyType>NEVER</pickEarlyType>

            </forager>

          </localSearch>

        </solver>

      </solverBenchmark>

      <solverBenchmark>

        <name>Entity tabu</name>

        <solver>

          <localSearch>

            <selector>

              <moveFactoryClass>org.drools.planner.examples.nqueens.solver.move.factory.RowChangeMoveFactory</moveFactoryClass>

            </selector>

            <acceptor>

              <planningEntityTabuSize>5</planningEntityTabuSize>

            </acceptor>

            <forager>

              <pickEarlyType>NEVER</pickEarlyType>

            </forager>

          </localSearch>

        </solver>

      </solverBenchmark>

      <solverBenchmark>

        <name>Value tabu</name>

        <solver>

          <localSearch>

            <selector>

              <moveFactoryClass>org.drools.planner.examples.nqueens.solver.move.factory.RowChangeMoveFactory</moveFactoryClass>

            </selector>

            <acceptor>

              <planningValueTabuSize>5</planningValueTabuSize>

            </acceptor>

            <forager>

              <pickEarlyType>NEVER</pickEarlyType>

            </forager>

          </localSearch>

        </solver>

      </solverBenchmark>

    </plannerBenchmark>

This `PlannerBenchmark` will try 3 configurations (1 move tabu, 1 entity tabu and 1 value tabu) on 2 data sets (32 and 64 queens), so it will run 6 solvers.

Every `solverBenchmark` entity contains a solver configuration (for example with a local search solver phase) and one or more `inputSolutionFile` elements. It will run the solver configuration on each of those unsolved solution files. A `name` is optional and generated if absent.

The common part of multiple `solverBenchmark` entities can be extracted to the `inheritedSolverBenchmark` entity, but that can still be overwritten per `solverBenchmark` entity. Note that inherited solver phases such as `<constructionHeuristic>` or `<localSearch>` are not overwritten but instead are added to the head of the solver phases list.

You need to specify a `benchmarkDirectory` (relative to the working directory). The best solution of each `Solver` run and a handy overview HTML file will be written in that directory.

### 11.2.3. ProblemIO: input and output of Solution files

#### 11.2.3.1. `ProblemIO` interface

The benchmarker needs to be able to read the input files to contain a `Solution` write the best `Solution` of each benchmark to an output file. For that it uses a class that implements the `ProblemIO` interface:

    public interface ProblemIO {

        String getFileExtension();

        Solution read(File inputSolutionFile);

        void write(Solution solution, File outputSolutionFile);

    }

#### 11.2.3.2. `XStreamProblemIO`

By default, a `XStreamProblemIO` instance is used and you just need to configure your `Solution` class as being annotated with XStream:

        <problemBenchmarks>

          <xstreamAnnotatedClass>org.drools.planner.examples.nqueens.domain.NQueens</xstreamAnnotatedClass>

          <inputSolutionFile>data/nqueens/unsolved/unsolvedNQueens32.xml</inputSolutionFile>

          ...

        </problemBenchmarks>

However, your input files need to have been written with a `XStreamProblemIO` instance.

#### 11.2.3.3. Custom `ProblemIO`

Alternatively, you can implement your own `ProblemIO` implementation and configure it with the `problemIOClass` element:

        <problemBenchmarks>

          <problemIOClass>org.drools.planner.examples.machinereassignment.persistence.MachineReassignmentProblemIO</problemIOClass>

          <inputSolutionFile>data/machinereassignment/input/model_a1_1.txt</inputSolutionFile>

          ...

        </problemBenchmarks>

### 11.2.4. Warming up the hotspot compiler

Without a warm up, the results of the first (or first few) benchmarks are not reliable, because they will have lost CPU time on hotspot JIT compilation (and possibly DRL compilation too).

The avoid that distortion, the benchmarker can run some of the benchmarks for a specified amount of time, before running the real benchmarks. Generally, a warm up of 30 seconds suffices:

    <plannerBenchmark>

      ...

      <warmUpSecondsSpend>30</warmUpSecondsSpend>

      ...

    </plannerBenchmark>

## 11.3. Summary statistics

### 11.3.1. Best score summary

Several summary statistics of the `PlannerBenchmark` run will be written in the `benchmarkDirectory`. Here is an example of a summary statistic:

Figure 11.1. Best score summary statistic

## 11.4. Statistics per data set (graph and CSV)

The benchmarker supports outputting statistics as graphs and CSV (comma separated values) files to the `benchmarkDirectory`.

To configure graph and CSV output of a statistic, just add a `problemStatisticType` line:

    <plannerBenchmark>

      <benchmarkDirectory>local/data/nqueens/solved</benchmarkDirectory>

      <inheritedSolverBenchmark>

        <problemBenchmarks>

          ...

          <problemStatisticType>BEST_SOLUTION_CHANGED</problemStatisticType>

          <problemStatisticType>CALCULATE_COUNT_PER_SECOND</problemStatisticType>

        </problemBenchmarks>

        ...

      </inheritedSolverBenchmark>

      ...

    </plannerBenchmark>

Multiple `problemStatisticType` elements are allowed. Some statistic types might influence performance noticeably. The following types are supported:

### 11.4.1. Best score over time statistic (graph and CSV)

To see how the best score evolves over time, add `BEST_SOLUTION_CHANGED` as `a problemStatisticType`.

Figure 11.2. Best score over time statistic

The best score over time statistic is very useful to detect abnormalities, such as score traps.

## Note

Don't be fooled by the simulated annealing line in this graph. If you give simulated annealing only 5 minutes, it might still be better than 5 minutes of tabu search. That's because this simulated annealing implementation automatically determines its velocity based on the amount of time that can be spend. On the other hand, for the tabu search, what you see is what you'd get.

### 11.4.2. Calculate count per second statistic (graph and CSV)

To see how fast the scores are calculated, add `CALCULATE_COUNT_PER_SECOND` as a `problemStatisticType`.

Figure 11.3. Calculate count per second statistic

## Note

The initial high calculate count is typical during solution initialization. In this example, it's far easier to calculate the score of a solution if only a handful exams have been added, in contrast to all of them. After those few seconds of initialization, the calculate count is relatively stable, apart from an occasional stop-the-world garbage collector disruption.

### 11.4.3. Memory use statistic (graph and CSV)

To see how much memory is used, add `MEMORY_USE` as `a problemStatisticType`.

Figure 11.4. Memory use statistic

- PrevChapter 10. Evolutionary algorithms
- Top of page
- Front page
- NextChapter 12. Repeated planning