JUNITPP
Extending JUNIT to Integration, System, Load and Stress Testing
DI Siegfried GOESCHL
IT Serv GmbH
Vienna, AUSTRIA
Abstract
This paper
describes an extension of the JUNIT 3.7 test framework called JUNITPP to
facilitate integration, system, stress and load testing. The extension provides
a test data repository, command line argument passing and an extended
TestRunner with built-in multi-threading support.
The JAVA unit test framework JUNIT is extensively used in the JAVA community due to its elegance of design and ease of use. Consequently the usage is extended beyond the original scope to cover integration, system, load and stress testing. Extending the scope naturally results in shortcomings such as the lack of a test data repository which stems from the original design. The absence of a test data repository results in hard-coded test data which increases the maintenance costs of testing database-driven systems. This fact should not be underestimated since the test code should have the same quality requirements as the application code. It is not uncommon to ship the regression test suite as part of the deliverables to increase customer confidence and to supplement the documentation. To overcome this and other limitations the JUNIT test framework was extended to provide a test data repository, command line arguments and an improved TestRunner supporting a built-in repetition counter and multithreading on the command line.
The main improvement of JUNITPP is the provision of a test data repository accessed by subclassing junit.extensions.ConfigurableTestCase.
public class FooTest extends
ConfigurableTestCase {
public FooTest(String name) {
super(name);
}
public void testFoo() {
String stringValue = getString("key1");
Boolean booleanValue = getBoolean("key2");
Integer integerValue = getInteger("key3");
Double doubleValue = getDouble("key4");
}
}
Figure 1 - Using the test data repository for a simple test
The class ConfigurableTestCase uses java.util.Properties to load the test data. The loading of the property file is triggered in the constructor of ConfigurableTestCase. For a master test suite which only invokes other test suites (usually AllTests) this strategy doesn’t work since no constructor is invoked. In this case loading the properties file has to be done manually by invoking ConfigurationFactory.init(AllTests.class) in the suite() method.
public class AllTests extends
ConfigurableTestCase {
public AllTests(String name) {
super(name);
}
public static Test suite() {
// load property file
ConfigurationFactory.init(AllTests.class);
TestSuite suite = new TestSuite();
suite.addTest(FooTest.class);
suite.addTest(BarTest.class);
return suite;
}
}
Figure 2 Using the test data repository for a master test suite
How is the properties file found when an instance of ConfigurableTestCase is executed? It is assumed that the property file has the same name as the class file but a different extension, i.e. the property file for the FooTest.class would be named FooTest.ini. The property file is either looked up in the current directory or in a list of directories such as “src”, “src/java”, “src/test” and “src/test/java”.
In some projects this approach might not work (e.g. using a single property file for all test suites) therefore the system property ‘junit.conf’ either defines a property file or a starting directory for the search.
java
–Djunit.conf=MyFooTest.conf junit.swingui.TestRunner FooTest
java
–Djunit.conf=./test/data junit.swingui.TestRunner FooTest
Figure 3 – Specifying a test data repository
How is an entry in the property file defined? To access the test data repository the class name, the name of the test case and the property name are concatenated to generate the key in the following order:
· fully qualified class name + test name + property name
· class name without package name + test name + property name
· class name without package name + property name
· property name
This implementation allows the reuse of test data definitions shared by one or
more test suites, e.g. the name of the sever used for testing a client/server
application with multiple test suites.
#
FooTest.conf
foo.FooTest.testFoo.key1=XYZ
FooTest.testFoo.key2=true
FooTest.key3=9999
Key4=3.1415927
Figure
4 - Content of a
test data repository
In the case of a master test suite the corresponding property file contains references to the property files of the contained test suites, which are loaded recursively.
#
AllTests.conf
.default.0=FooTest.conf
.default.1=BarTest.conf
Figure
5 - Content of a
test data repository for a master suite
What happens if a different type of test data repository already exists, e.g. in a relational database? In JUNIT++ the implementation of test data repository is determined and instantiated at runtime. Hence it is possible to provide a different implementation of a test data repository by defining the implementation class at the command line:
java
-Djunit.conf.class=xyz.JDBCTestProperties junit.extensions.PPTestRunner
foo.bar.AllTests
Figure 6 - Using a different implementation of a test data repository
After a successful test execution one might reuse the test suite to gather performance data either by simply measuring the execution time or profiling the application. The profiling gives a good indication of how much time and resources are spent in an ORB runtime library or a Servlet engine. It is necessary to fine-tune the number of test execution to find a balance between the effect of one-time initializations distorting the result and time required to generate the profiling data. By using junit.extensions.PPTestRunner the number of test repetitions can be defined on the command line:
java
junit.extensions.PPTestRunner –r 10 foo.bar.AllTests
Figure 7 - Starting PPTestRunner with 10 repetitions
If the profiling of a client/server application is done on one computer it will become overloaded. In this case a delay of the test case invocations can be defined on the command line:
java
junit.extensions.PPTestRunner –r 10 -w 100 foo.bar.AllTests
Figure 8 – Executing tests with 10 repetitions and 100 ms invocation delay
The next step might be overloading the server until it crashes due to race conditions and/or resource shortage. This can be accomplished by invoking one or more test suites using multiple threads, repetition counters and test case invocation delays to simulate a more realistic user behaviour.
java
junit.extensions.PPTestRunner –r 1000 –s 10 foo.bar.AllTests
Figure 9 – Server crash test with a client executing 1000 times with 10 threads simultaneously
Some components require command line parameters for initialization. JUNITPP provides a uniform way of defining application specific command line parameters by passing them as a system property. Some development environments are unable to process quoted arguments when invoking the debugger. This is why multiple arguments can be separated by the ‘%’ sign.
java
–Djunit.params="-c foo.ext" junit.textui.TestRunner FooTest
java
–Djunit.params=-c%foo.ext junit.textui.TestRunner FooTest
Figure 10 - Defining command line arguments
Sometimes passing arguments via command line is clumsy, e.g. initialising a JDBC connection for multiple test suites. In this case a ConfigurableTestCase can be used, where the parameters are stored in a corresponding property file.
public class JDBCTestSetup extends
ConfigurableTestSetup {
static private Connection connection = null;
public JDBCTestSetup() {}
public void setUp() {
connection = DriverManager.getConnection(
getString("url"),
getString("user"),
getString("password");
}
public void tearDown() {
connection.close();
}
public Connection getConnection() {
return connection;
}
}
Figure 11 - Using a ConfigurableTestSetup
If a test suite does a lot of printing, it is useful to get a more verbose output of a test run. This can be accomplished by turning on the verbose mode of PPTestRunner.
Figure 12 – Using the verbose mode
JUNITPP offers a set of functions necessary to extend the scope of usage from unit testing to integration, system, load and stress testing. These extensions include a test data repository for composite test suites, command line argument support, a ConfigurableTestSetup and an extended TestRunner.
The usage of JUNITPP makes it possible to separate test data from test code, which is essential for testing database-driven applications. Omitting this separation results in hard-coded test data, which increase the test maintenance costs and decrease the acceptance of regression tests.
The ease of generating an arbitrary load under various conditions encourages developers to use stress tests for components at a very early stage of development which in turn increases the reliability of the final product. This observation is particularly relevant for distributed systems because it might require an expert’s knowledge to trace a failure back to a single component.