Automated performance testing using JMeter and Maven - Directory structure for the automated test

Geschrieben von: George Barnett Donnerstag, den 22. Oktober 2009 um 19:20 Uhr

News Developer

The structure for the automated test is slightly different as you can see below. Create a new project in your SCM and create the directories listed.

The main difference is that we're using the test directory instead of main. This is because the main purpose of this project is to run tests instead of build an artefact.

./pom.xml
./src
./src/test
./src/test/resources
./src/test/resources/jprofilerconfig.xml
./src/test/resources/dataset-sanity1
./src/test/resources/dataset-sanity1/config.xml
./src/test/resources/dataset-sanity1/repository
./src/test/resources/dataset-sanity1/repository/filePaths.csv
./src/test/resources/dataset-sanity1/repository/repoNames.csv
./src/test/resources/dataset-sanity1/repository/repoPaths.csv
./src/test/resources/dataset-sanity1/search
./src/test/resources/dataset-sanity1/search/complexTerms.csv
./src/test/resources/dataset-sanity1/search/eyeqlTerms.csv
./src/test/resources/dataset-sanity1/search/simpleTerms.csv

Write a pom.xml for the automated test

Resource: pfe-automation-pom.xml

It's best to start out by working out what actions (and in which order) you will need to take to test your application. In this example, here are the steps the test will perform:

1. Prepare the FishEye data directory using the chosen data to load
2. Prepare the Jprofiler config
3. Download and extract the FishEye application
4. Start FishEye
5. Run a JMeter test using the defined CSV data
6. Stop FishEye
7. Run reporting on the JMeter outputs and generate a site directory for upload

It's important to define what order this work should occur in before starting so you know what you are aiming for. I'm not going to annotate the whole pom.xml since it is several hundred lines long. I'll go over the important parts which you may not have seen before - I would suggest opening a copy locally before proceeding.

POM: Define build profiles

There will be many tasks to perform and settings to choose when running the test. Some of these tasks will need to happen every time (for example, setting up a database) and others will only need to happen when you are testing something specific (such as adding a code profiler). Maven provides the concept of Profiles (not to be confused with code profiling) which can be used to add and remove plugins to your test harness.

In this example, I will set up two profiles. By default, the information contained in them will not be used but when we do want to use it, it can be added to the test with a simple switch.

POM: Import Dependencies

<dependency>
  <groupId>com.atlassian.performance.fisheye</groupId>
  <artifactId>performance-test</artifactId>
  <type>zip</type>
  <version>${project.version}</version>
</dependency>

The JMeter performance test artefact we packaged up earlier is a dependency in this test. Using the maven-dependency-plugin this artefact will be extracted into the target/test-classes/ directory for use.

The maven-dependency-plugin is also used to download and extract the application that's going to be tested.

The Maven exec plugin is used to start FishEye using the exec:exec goal. FishEye is started at the pre-integration-test phase so it's ready for use with the JMeter goals which run at integration-test phase. If your application runs inside an application server such as Tomcat or Jetty, you should use the Maven Cargo plugin to start and stop your application.

POM: Plugins - JMeter configuration

The JMeter test is run by the Maven Chronos plugin which is responsible for running JMeter. There a few settings which you need to pay attention to:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>chronos-maven-plugin</artifactId>
  <version>1.0</version>
  <configuration>
    <jmeterhome>${jmeter.home}</jmeterhome>
    <historydir>${history.directory}</historydir>
    <heap>756m</heap>

jmeterhome

Where JMeter is installed but the base directory - JMeter should be in bin/jmeter under this directory. It's worth nothing that the plugin does not start jmeter via the startup script but rather invokes the jar's directly.

historydir

This variable should be a unique directory for the build. Since you're going to eventually have several profiles with different datasets and JVM versions, it should take this into account. I use the following format:

history.directory / product / product.version-java.version-data.version

heap

It's important to give JMeter enough memory. This is especially true if your test uses multiple Listeners. As you will see later when I cover reporting, the default report will include a graph showing Garbage Collection. This graph is actually JMeters garbage collection, but it's good to keep an eye on it. If JMeter spends too much time doing GC then the test results will be skewed and useless.

The settings in the attached pom.xml include some tweaks to the Garbage Collection, new size and survivor ratio.

POM: Plugins - JMeter System Properties

These variables are initially set as global configuration items but are then overridden in the profile for the dataset used. The variables here will be passed to JMeter using the -Dvariable.name=value command line switch.

<sysproperties>
  <property>
    <name>script.runtime</name>
    <value>${script.fixedload.runtime}</value>
  </property>
  <property>
    <name>script.base</name>
    <value>${project.build.directory}/performanceTest</value>
  </property>
  <property>
    <name>resource.dir</name>
    <value>${dataset.location}</value>
  </property>
</sysproperties>

JMeter accepts external variable input using the P function, eg: ${__P(script.runtime,1800)}

Using sysproperties in the pom, it's possible to override the variables inside the JMeter test. The most useful reason for this doing is to change the location of the resources which contains the CSV files which define which pages to sample directory. In the above example, the dataset.location variable is modified by the chosen profile in the pom and this is passed to JMeter, thus allowing the override of the resource directory used for testing.

POM: Plugins - JMeter test executions

<executions>
  <execution>
  <id>71_fixedload-test</id>
    <configuration>
      <dataid>fixedloadtest</dataid>
       <input>${project.build.directory}/performanceTest/jmeter-test-fixedload.jmx</input>
    </configuration>
    <goals>
      <goal>jmeter</goal>
      <goal>check</goal>
      <goal>savehistory</goal>
    </goals>
  </execution>
</executions>

The sysproperties shown earlier will be applied to all exections of the plugin. There are however two config variables above must be unique for each execution however:

dataid

This provides a unique name the test data output will be known as. It will be used to save files in the reports later and the savehistory goal will use this to save the data for this execution.

input

This is the .jmx performance test script to run.

Chronos Build Goals

POM: Reporting

<reporting>
  <plugins>
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>chronos-maven-plugin</artifactId>
      <version>1.0-atlassian-1</version>
      <configuration>
        <historydir>${history.directory}</historydir>
      </configuration>
      <reportSets>
        <reportSet>
          <id>fixedloadreport</id>
          <configuration>
            <dataid>fixedloadtest</dataid>
            <reportid>jmeter-fixedload-report</reportid>
            <title>JMeter Fixed Load Test Report</title>
            <description>
              <![CDATA[Fixed Load Test Report]]>
            </description>
          </configuration>
          <reports>
            <report>report</report>
            <report>historyreport</report>
          </reports>
        </reportSet>
      </reportSets>
    </plugin>
  </plugins>
</reporting>

The reporting lifecycle produces graphs, one of which I've shown as an example at the top of the page. You should include one reportSet for each execution that was defined in the build part of the Chronos plugin. The dataid of the reportSet should match the relevant test execution dataid.

POM: Profiles

In order to control the dataset, I've added a profile for each piece of data I will be loading into FishEye. At the moment, I have only one, but making another will be easy with profiles. Most of the profile will be setting up variables which will be used elsewhere in the pom. For example, the dataset.location variable will be passed into JMeter using a sysproperty to set the resource base directory.

<profile>
  <id>dataset-sanity1</id>
  <properties>
    <repository.zipfile>${performance.datadir}/sanity1.zip</repository.zipfile>
    <repository.location>${repository.basedir}/sanity1</repository.location>
    <dataset.location>${dataset.basedir}/dataset-sanity1</dataset.location>
    <dataset.version>sanity1</dataset.version>
    <fisheye.config>${dataset.location}/config.xml</fisheye.config>
  </properties>
</profile>

Run the Tests!

Once your pom is complete, you can run the test using the mvn command. Don't forget to have any profiles you need as part of the command:

The test has now run successfully. In the target directory you'll find a site directory which contains HTML and images to make up a build report. This will be uploaded to the URL specified in the distributionManagement section of the pom.

Automate the tests using Bamboo

Bamboo is a continuous integration server made by Atlassian and is perfect for running the tests regularly.

For information on creating a new build plan in Bamboo, see the document Creating a build plan in the online documentation for Bamboo.

In closing..

With this information, you can start to make a real difference to your development cycle. You'll find performance regressions faster and be on you way to code bliss. If you're interested in some of the processes which you can use to better integrate this cycle, see Andrew Prentice's blog on improving the QA cycle in You don't have to be Agile to be agile.

Learn more about software testing in an agile development environment