
  JacORB Test Suite Documentation
  ===============================

This directory tree holds the JacORB regression test suite.  It is not
included in most releases of JacORB, because the test suite is not
necessary for using the ORB.  The suite is always available from
JacORB's public CVS repository, for everybody who wishes to run the
tests or to improve them.  Many of the test cases are also good
examples that illustrate how to use certain features of JacORB.

Although the test suite is based on JUnit in a rather straightforward
way, there are a few special mechanisms built into the regression test
framework.  These include:

  * a framework for testing the IDL compiler against arbitrary IDL
    source, checking for proper error handling and verifying whether
    the generated Java code can indeed be compiled.

  * the ability to run end-to-end client/server tests.  In these
    tests, the Java process that executes the tests under JUnit acts
    as the client, and it automatically spawns a separate server
    process with which the client side communicates.

  * the ability to use different ORB versions on the client side and
    the server side.  This is achieved through a launching framework
    that can start client and server processes against arbitrary
    versions of JacORB, installed anywhere in the file system.  It is
    also possible to run some of the tests against a TAO ORB on the
    server side, provided that the corresponding test server code has
    been re-implemented in C++ using TAO.

  * the ability to enable or disable certain test cases, based on the
    ORB versions used in the tests.  This way, some test cases can be
    skipped automatically when using an older ORB that does not
    support the features tested by those test cases.  To achieve this,
    test cases can be marked with special Javadoc comments that
    indicate which ORB versions they apply to.

The test suite can also be executed under the EMMA coverage tool,
which measures and reports the amount of coverage within the JacORB
code that was achieved during the test.


Building the Test Suite
-----------------------

We assume that you obtain the regression suite code by checking all of
JacORB out of the public CVS repository.  This means that you have a
functioning, current JacORB version within which the test suite itself
is located.  By default, the test suite uses this JacORB version to
run the tests, and it refers to this version as the "cvs" version of
JacORB.

To build the tests, first build the surrounding JacORB code by
following the standard installation instructions in the file
JACORB_HOME/doc/install.html.  Then change directory to
JACORB_HOME/test/regression (in the following, this directory will be
called REGRESSION_HOME).  Type:

  ant build-all

This will compile all the IDL and Java code in the regression suite,
and make it ready to run (see below for instructions on that).

You can also use other versions of JacORB for running the tests.  To
do this, download those versions separately, install and build them
anywhere in the file system.  Define those versions in the file
test.properties in REGRESSION_HOME/resources.  There are comments in this file
that explain how to define other JacORB versions.


Running the Tests
-----------------

You can run the tests either from the command line or from within
Eclipse.  Using the command line is the "standard" way to run the
tests, which is most flexible in terms of configuring the execution
environment, but running from within Eclipse is usually more
convenient for isolating individual failures and debugging issues.

a) Running the tests from the command line

The standard way to run the tests from the command line is the ant
target

  ant run-all

This target compiles and runs all the tests. It's possible to skip
the compilation step by passing the property NO_BUILD to ant. This isn't 
necessary as the various ant tasks are able to figure
that out by themselves if they need to do anything.

  ant run-all -DNO_BUILD=true

The test protocol is written both to the terminal and to a file
that's named similiar to the current test. The target run-all would 
cause the creation of the logfile TEST-org.jacorb.test.dii.AllTest.txt.
By default the testresults are written to a newly created directory under
REGRESSION_HOME/output.
The new directory is named with a timestamp in the format
YYYY-MM-DD.HH-MM-SS, this is a unique name that's intended to make it
easy to refer to test results later. If you don't want a new directory for 
every testrun you can set the property jacorb.test.outputfile.testname. This
causes all output to be put into the directory REGRESSION_HOME/output directly. Note
that the contents of that directory are deleted before the tests are run.

  ant run-all -Djacorb.test.outputfile.testname=true

The output logfile is quite verbose.  To determine the overall
test result, search for the line that starts with the words "Tests
run:".  For example:

  Tests run: 932, Failures: 7, Errors: 1, Time elapsed: 333.253 sec

In this summary, a "Failure" means that a test assertion was not met,
and an "Error" means that an unexpected exception occured at some
point while a test was executed.

You will find many stack traces, exception dumps and error reports in
report.txt.  Most of them (hopefully, all of them) are provoked
exceptions where we check whether JacORB handles various error
conditions properly.  They do not indicate that anything went wrong
running the tests.

To find test cases that actually failed, search for the string FAILED
on a line by itself in report.txt.  The name of the test case that
failed appears before this line, and you can use this name to search
for the code of the failed test case in REGRESSION_HOME/src.  However,
usually it's more effective to run some of the tests from within
Eclipse, as explained in the next section, and investigate the results
there.

There are various properties you can define when running the tests to
change the behaviour (for example ant -Djacorb.test.coverage=on
run-all-no-build).  The most important of these properties are:

  jacorb.test.junitreport.html.process
                                create html reports in addition to the simple .txt
                                format. the html reports are put into the directory
                                specified by the property 
                                jacorb.test.junitHtmlDir (defaults to OUTDIR/html).

  jacorb.test.ssl=true/false    run all client/server tests with SSL enabled

  jacorb.test.imr=true/false    run all client/server tests via the
                                implementation repository

  jacorb.test.coverage=true/false   
                                collect coverage data while running
                                (see below for explanations)

  jacorb.test.server.version=ID use JacORB version ID for the server side
                                (ID must be defined in test.properties)
         
  jacorb.test.client.version=ID use JacORB version ID for the client side
                                (ID must be defined in test.properties)

  jacorb.test.verbose=true/false 
                                additional informational output by the regression
                                framework and the testcases.

b) Running the tests from Eclipse

You can run any test as a normal JUnit test from within Eclipse, which
allows you to track down much more effectively which test cases
failed, and why.  To do this, import JacORB as an Eclipse project, as
described in the HOWTO document in JACORB_HOME/doc/HOWTO.  You can
then run any of the JacORB tests by selecting the TestCase
class and clicking "Run As...".  Run it as a "JUnit Test Case".  To
make the tests run properly, you must likely define the property
jacorb.home as a VM argument: -Djacorb.home=YOUR_JACORB_HOME.  No
other definitions should be necessary.

The root TestSuite class that includes all the other tests is
org.jacorb.test.AllTest.  You can drill down into the package
hierarchy under org.jacorb.test and choose any other TestSuite as
well.  The classes named "AllTest" are good candidates.

When you run the tests this way, you lose the ability to select
different JacORB versions for the client side (because the tests will
always run with the Build Path that is set up within Eclipse for the
client side).  Different server versions can be selected by defining
the property jacorb.test.server.version, as usual (see above).

Likewise, no output directory will be created for the test run, and no
logfile will be written.

You can use Eclipse's normal debugging facilities to debug client side
issues.  There is no support for debugging server side issues (yet),
because the server is started as a separate Java process that is not
under the control of Eclipse.  To debug server side issues, use print
statements in the code -- the output of such print statements should
appear on the console during the test run.

If you want to run the tests in the same configuration as from the
command line, but from within Eclipse, run the main method of the
class org.jacorb.test.common.launch.TestLauncher.  This allows you to
change the client-side JacORB version, and to write a log to the usual
output directory, but you will not be able to inspect the test results
using Eclipse's facilities.  From Eclipse's point of view, the test
run is just another Java process in this case.


Getting Coverage Reports  
------------------------

You can obtain coverage data for test runs using the EMMA coverage
tool, which is integrated into the regression suite.  Obtaining
coverage is a three-step process: (1) instrumenting the JacORB code,
(2) running the tests and collecting coverage data, (3) generating a
coverage report from the data in HTML.

To instrument the code, go to REGRESSION_HOME and type

  ant instrument

This is usually done during the build-all target if you haven't
disabled building by passing the NO_BUILD property.

This will create a new directory JACORB_HOME/classes-instrumented that
contains the instrumented code, and a metadata file
JACORB_HOME/coverage.em.  

To collect coverage data during a test run, define the property
jacorb.test.coverage=on, for example:

  ant -Djacorb.test.coverage=true run-all

After the tests have completed ant will generate html reports from the
coverage data and put them into the directory specified by the property
jacorb.test.coverage.dir (defaults to OUTDIR/coverage).

In the output directory, you will then find three report directories,
coverage-client, coverage-server, and coverage-combined.  These
directories contain reports for the client side, the server side, and
both of them combined, respectively.  (Tests that are not
client/server tests count on the client side.)  You can view these
reports by pointing a web browser to the file index.html in each of
the subdirectories, and drill down into packages and classes as you
like.

Note: There is currently an issue with the test
org.jacorb.test.bugs.bugjac330, which often corrupts the coverage data
because of a race condition.  For the time being, comment out this
test in org.jacorb.test.bugs.AllTest when obtaining coverage
information.  This issue will be corrected soon.


Adding new tests
----------------

The following steps are necessary to add a new set of JUnit tests to
the JacORB test suite.

1. Write the tests using the JUnit framework. The sourcefile should be
   named XYZTest.java (replace XYZ with an appropiate name). Tests
   should be packaged using the same packaging as the source that is
   being tested, but replacing 'org.jacorb' with 'org.jacorb.test'.
   Any helper classes that can be used by tests in different packages
   should be put in the package 'org.jacorb.test.common'.  All source
   code in the test hierarchy can be found under the 'src' directory.

   For example, the DynAny source is in the package 'org.jacorb.orb.dynany'.
   Therefore, the JUnit tests that test the DynAny source are in the package
   'org.jacorb.test.orb.dynany'.  Many tests in the orb package and elsewhere
   use a special client/server setup; this can be found under
   org.jacorb.test.common.

2. Create a Java source file named 'AllTest.java' that will run all JUnit
   tests within the package and each of its subpackages.  An example of this
   file can be found in each package in the test hierarchy.  The AllTest
   file at the root of the test hierarchy (in the package 'org.jacorb.test')
   will run all the tests in the repository.  It does this by calling on to
   the AllTest test suite in each of its subpackages, which in turn call on to
   the AllTest test suites in each of their subpackages, until all tests are
   executed.  If adding tests to a package that already contains this file then
   no modifications to the existing file are required.

3. Modify the AllTest file in the package immediately above the package where
   the tests are being added.  This file needs to include the test suite of
   the AllTest file in the newly added package of tests.  If adding tests to
   a package that already contains an AllTest file then this modification is
   not required.

   IDL
   ---

1. Add any IDL files to the 'idl' directory in the test repository.  The
   'pragma prefix' directive should be set to 'jacorb.org'.

   Ant XML
   -------

1. The following modifications are needed for each new IDL file added to the
   test repository:

   Copy and modify the target named 'idl-tests'.  This target compiles the
   Tests.idl file.  To change this target to compile a different IDL file,
   set the values of the three properties in the target as follows:
      includes = Comma- or space-separated list of files 
                 (may be specified using wildcard patterns)
                 that must be included. The files are looked
                 up relative to the directory test/regression/idl
      ami      = true/false. Specify if AMI classes should be generated
                 for the current IDL files. This is optional
                 and defaults to false.

   Rename this target to 'idl-<name>' where <name> refers to the name of the
   IDL file (excluding the .idl extension and beginning with a lower case
   character).

   Modify the target named 'compile-idl' to depend on the newly added
   target.  This modification involves changing the depends attribute
   to contain the newly added target that was created in the above
   step.
