<?xml version="1.0" encoding="UTF-8"?>
<chapter id="testing">
  <title>Testing</title>

  <section id="testinggoals">
    <title>Overall goals</title>
    <para>The overall goals of our test environment is to execute tests that ensures that
    we have full coverage of the JCA specification as well as our implementation.</para>

    <para>The full test suite is executed using</para>

    <programlisting>
ant test
    </programlisting>

    <para>A single test case can be executed using</para>

    <programlisting>
ant -Dmodule=embedded -Dtest=org.jboss.jca.embedded.unit.ShrinkWrapTestCase one-test
    </programlisting>

    <para>where <code>-Dmodule</code> specifies which module to execute the test case in. This parameter
      defaults to <code>core</code>. The <code>-Dtest</code> parameter specifies the test case itself.</para>

    <para>You can also execute all test cases of a single module using</para>

    <programlisting>
ant -Dmodule=embedded module-test
    </programlisting>

    <para>where <code>-Dmodule</code> specifies which module to execute the test cases in. This
      parameter defaults to <code>core</code>.</para>

    <para>The build script does not fail in case of test errors or failure.</para>

    <para>
      You can control the behavior by using the <code>junit.haltonerror</code> and <code>junit.haltonfailure</code> 
      properties in the main <code>build.xml</code> file. Default value for both is <code>no</code>.
    </para>

    <para>
      You can of course change them statically in the <code>build.xml</code> file or temporary using <code>-Djunit.haltonerror=yes</code>.
      There are other <code>jnuit.*</code> properties defined in the main <code>build.xml</code> that can be controlled in the same
      way.
    </para>

    <section id="spectest">
      <title>Specification</title>
      <para>The purpose of the specification tests is to test our implementation against the
      actual specification text.</para>
      <para>Each test can only depend on:</para>
      <itemizedlist spacing="compact">
         <listitem>
            <para>The official Java Connector Architecture API (javax.resource)</para>
         </listitem>
         <listitem>
            <para>Interfaces and classes in the test suite that extends/implements 
            the official API</para>
         </listitem>
      </itemizedlist>
      <para>The test cases should be created in such a way such that they are easily identified
        by chapter, section and paragraph. For example:</para>
      <programlisting>
org.jboss.jca.core.spec.chaper10.section3
      </programlisting>
    </section>

    <section id="jbinterfacetest">
      <title>IronJacamar specific interfaces</title>
      <para>The purpose of the IronJacamar specific interfaces tests is to test our specific interfaces.</para>
      <para>Each test can depend on:</para>
      <itemizedlist spacing="compact">
         <listitem>
            <para>The official Java Connector Architecture API (javax.resource)</para>
         </listitem>
         <listitem>
            <para>The IronJacamar specific APIs (org.jboss.jca.xxx.api)</para>
         </listitem>
         <listitem>
            <para>Interfaces and classes in the test suite that extends/implements these APIs</para>
         </listitem>
      </itemizedlist>
      <para>The test cases lives in a package that have a meaningful name of the component it tests. For example:</para>
      <programlisting>
org.jboss.jca.core.workmanager
      </programlisting>

      <para>These test cases can use both the embedded JCA environment or be implemented as standard POJO based
        JUnit test cases.</para>

    </section>

    <section id="jbimpltest">
      <title>IronJacamar specific implementation</title>
      <para>The purpose of the IronJacamar specific implementation tests is to test our specific implementation.
      These tests should cover all methods are not exposed through the interface.</para>
      <para>Each test can depend on:</para>
      <itemizedlist spacing="compact">
         <listitem>
            <para>The official Java Connector Architecture API (javax.resource)</para>
         </listitem>
         <listitem>
            <para>The IronJacamar specific APIs (org.jboss.jca.xxx.api)</para>
         </listitem>
         <listitem>
            <para>The IronJacamar specific implementation (org.jboss.jca.xxx.yyy)</para>
         </listitem>
         <listitem>
            <para>Interfaces and classes in the test suite</para>
         </listitem>
      </itemizedlist>
      <para>The test cases lives in a package that have a meaningful name of the component it tests. For example:</para>
      <programlisting>
org.jboss.jca.core.workmanager
      </programlisting>

      <para>These test cases can use both the embedded JCA environment or be implemented as standard POJO based
        JUnit test cases.</para>

    </section>
  </section>

  <section id="style">
    <title>Testing principle and style</title>
    <para>
      Our tests follows the Behavior Driven Development (BDD) technique. In BDD you focus on specifying the behaviors
      of a class and write code (tests) that verify that behavior.
    </para>
    <para>
      You may be thinking that BDD sounds awfully similar to Test Driven Development (TDD). 
      In some ways they are similar: they both encourage writing the tests first and to provide full coverage of the 
      code. However, TDD doesn't really provide a guide on which kind of tests you should be writing.
    </para>
    <para>
      BDD provides you with guidance on how to do testing by focusing on what the behavior of a class is supposed to be.
      We introduce BDD to our testing environment by extending the standard JUnit 4.x test framework with
      BDD capabilities using assertion and mocking frameworks.
    </para>
    <para>
      The BDD tests should
    </para>
    <itemizedlist spacing="compact">
      <listitem>
        <para>
          Clearly define <code>given-when-then</code> conditions
        </para>
      </listitem>
      <listitem>
        <para>
          The method name defines what is expected: f.ex. shouldReturnFalseIfMethodXIsCalledWithNullString()
        </para>
      </listitem>
      <listitem>
        <para>
          Easy to read the assertions by using <ulink url="http://code.google.com/p/hamcrest/">Hamcrest Matchers</ulink>
        </para>
      </listitem>
      <listitem>
        <para>
          Use <code>given</code> facts whenever possible to make the test case more readable. It could be the
          name of the deployed resource adapter, or using the 
          <ulink url="http://mockito.googlecode.com/svn/branches/1.8.0/javadoc/org/mockito/BDDMockito.html">
            BDD Mockito class</ulink> to mock the fact.
        </para>
      </listitem>
    </itemizedlist>
    
    <para>We are using two different kind of tests:</para>
    <itemizedlist spacing="compact">
      <listitem>
        <para>
          Integration Tests: The goal of these test cases is to validate the whole process of deployment, and
          interacting with a sub-system by simulating a critical condition.
        </para>
      </listitem>
      <listitem>
        <para>
          Unit Tests: The goal of these test cases is to stress test some internal behaviour by mocking classes
          to perfectly reproduce conditions to test.
        </para>
      </listitem>
    </itemizedlist>

    <section id="integration">
      <title>Integration Tests</title>
      <para>
        The integration tests simulate a real condition using a particular deployment artifacts packaged as 
        resource adapters.
      </para>

      <para>
        The resource adapters are created using either the main build environment or by using 
        <ulink url="http://community.jboss.org/wiki/ShrinkWrap">ShrinkWrap</ulink>. Using resource adapters
        within the test cases will allow you to debug both the resource adapters themself or the JCA container.
      </para>

      <para>
        The resource adapters represent the <citation>given</citation> facts of our BDD tests, 
        the deployment of the resource adapters represent the <citation>when</citation> phase, 
        while the <citation>then</citation> phase is verified by assertion.
      </para>

      <para>
        Note that some tests consider an exception a normal output condition using the JUnit 4.x 
        <code>@Exception(expected = "SomeClass.class")</code> annotation to identify and verify this situation. 
      </para>
    </section>

    <section id="unit">
      <title>Unit Tests</title>
      <para>
        We are mocking our input/output conditions in our unit tests using the 
        <ulink url="http://mockito.googlecode.com">Mockito</ulink> framework to verify class and method behaviors.
      </para>
      <para>An example:</para>
      <programlisting language="Java" role="JAVA">
@Test
public void printFailuresLogShouldReturnNotEmptyStringForWarning() throws Throwable
{
   //given
   RADeployer deployer = new RADeployer();
   File mockedDirectory = mock(File.class);
   given(mockedDirectory.exists()).willReturn(false);

   Failure failure = mock(Failure.class);
   given(failure.getSeverity()).willReturn(Severity.WARNING);

   List failures = Arrays.asList(failure);
   FailureHelper fh = mock(FailureHelper.class);
   given(fh.asText((ResourceBundle) anyObject())).willReturn("myText");
  
   deployer.setArchiveValidationFailOnWarn(true);
  
   //when
   String returnValue = deployer.printFailuresLog(null, mock(Validator.class), 
                                                  failures, mockedDirectory, fh);
  
   //then
   assertThat(returnValue, is("myText"));
}
      </programlisting>
      <para>
        As you can see the BDD style respects the test method name and using the 
        <code>given-when-then</code> sequence in order.
      </para>
    </section>
  </section>

  <section id="qa">
    <title>Quality Assurance</title>
    <para>In addition to the test suite the IronJacamar project deploys various
      tools to increase the stability of the project.</para>
    <para>The following sections will describe each of these tools.</para>

    <section id="checkstyle">
      <title>Checkstyle</title>
      <para>Checkstyle is a tool that verifies that the formatting of the source
        code in the project is consistent.</para>

      <para>This allows for easier readability and a consistent feel of the project.</para>

      <para>The goal is to have zero errors in the report. The checkstyle report is generated
        using</para>

      <programlisting>
ant checkstyle
      </programlisting>

      <para>The report is generated into</para>

      <programlisting>
reports/checkstyle
      </programlisting>

      <para>The home of checkstyle is located here: <ulink url="http://checkstyle.sourceforge.net/"/>.</para>

    </section>

    <section id="findbugs">
      <title>Findbugs</title>
      <para>Findbugs is a tool that scans your project for bugs and provides reports based on its
        findings.</para>

      <para>This tool helps lower of the number of bugs found in the IronJacamar project.</para>

      <para>The goal is to have zero errors in the report and as few exclusions in the filter as possible.
        The findbugs report is generated using</para>

      <programlisting>
ant findbugs
      </programlisting>

      <para>The report is generated into</para>

      <programlisting>
reports/findbugs
      </programlisting>

      <para>The home of findbugs is located here: <ulink url="http://findbugs.sourceforge.net/"/>.</para>

    </section>

    <section id="jacoco">
      <title>JaCoCo</title>
      <para>JaCoCo generates a test suite matrix for your project which helps you identify
        where you need additional test coverage.</para>

      <para>The reports that the tool provides makes sure that the IronJacamar project has the correct test coverage.</para>

      <para>The goal is to have as high code coverage as possible in all areas. The JaCoco report is
        generated using</para>

      <programlisting>
ant jacoco
      </programlisting>

      <para>The report is generated into</para>

      <programlisting>
reports/jacoco
      </programlisting>

      <para>The home of JaCoCo is located here: <ulink url="http://www.eclemma.org/jacoco/"/>.</para>

    </section>

    <section id="tattletale">
      <title>Tattletale</title>
      <para>Tattletale generates reports about different quality matrix of the dependencies within the project.</para>

      <para>The reports that the tool provides makes sure that the IronJacamar project doesn't for example have
        cyclic dependencies within the project.</para>

      <para>The goal is to have as no issues flagged by the tool. The Tattletale reports are
        generated using</para>

      <programlisting>
ant tattletale
      </programlisting>

      <para>The reports are generated into</para>

      <programlisting>
reports/tattletale
      </programlisting>

      <para>The home of Tattletale is located here: <ulink url="http://www.jboss.org/tattletale"/>.</para>

    </section>

  </section>

  <section id="performance">
    <title>Performance testing</title>
    <para>Performance testing can identify areas that needs to be improved or completely replaced.</para>

    <section id="jprofiler">
      <title>JProfiler</title>

      <para>Insert the following line in <code>run.sh</code> or <code>run.bat</code>:</para>

      <programlisting>
-agentpath:&lt;path&gt;/jprofiler6/bin/linux-x64/libjprofilerti.so=port=8849 
      </programlisting>

      <para>where the Java command is executed.</para>

      <para>The home of JProfiler is located here: <ulink url="http://www.ej-technologies.com/products/jprofiler/overview.html"/>.</para>

    </section>

    <section id="oprofile">
      <title>OProfile</title>

      <para>OProfile can give a detailed overview of applications running on the machine, including
        Java program running with OpenJDK.</para>

      <para>The home of OProfile is located here: <ulink url="http://oprofile.sourceforge.net"/>.</para>

      <section id="oprofile_install">
        <title>Installation</title>
        <para>Enable the Fedora debug repo:</para>
        <programlisting>
/etc/yum.repos.d/fedora.repo

[fedora-debuginfo]
name=Fedora $releasever - $basearch - Debug
failovermethod=priority
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&amp;arch=$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch
        </programlisting>

        <para>Install:</para>
        <programlisting>
yum install -y oprofile oprofile-jit
yum install -y yum-plugin-auto-update-debug-info
yum install -y java-1.6.0-openjdk-debuginfo
        </programlisting>
      </section>

      <section id="oprofile_running">
        <title>Running</title>
        <para>Insert the following line in <code>run.sh</code> or <code>run.bat</code>:</para>

        <programlisting>
-agentpath:/usr/lib64/oprofile/libjvmti_oprofile.so
        </programlisting>

        <para>for 64bit JVMs or </para>

        <programlisting>
-agentpath:/usr/lib/oprofile/libjvmti_oprofile.so
        </programlisting>

        <para>for 32 bit JVMs where the Java command is executed.</para>

        <para>Now execute:</para>
        <programlisting>
opcontrol --no-vmlinux
opcontrol --start-daemon
        </programlisting>

        <para>and use the following commands:</para>
        <programlisting>
opcontrol --start # Starts profiling
opcontrol --dump  # Dumps the profiling data out to the default file
opcontrol --stop  # Stops profiling
        </programlisting>

        <para>Once you are done execute:</para>
        <programlisting>
opcontrol --shutdown  # Shuts the daemon down
        </programlisting>

        <para>A report can be generated by:</para>
        <programlisting>
opreport -l --output-file=&lt;filename&gt;
        </programlisting>

        <para>Remember that this is system wide profiling, so make sure that only the services
          that you want included are running.</para>

        <para>More information is available at <ulink url="http://oprofile.sourceforge.net/doc/index.html"/>.</para>

      </section>

    </section>

    <section id="performance_test_suite">
      <title>Performance test suite</title>

      <para>
        IronJacamar features a basic performance test suite that tests interaction with
        a transaction manager.
      </para>

      <para>
        The test suite is executed by
      </para>
      <programlisting>
ant perf-test
      </programlisting>
      <para>
        which will run the tests, and output its data into the generated JUnit output.
      </para>

      <para>
        The setup of the performance test suite is controlled in the
      </para>
      <programlisting>
org.jboss.jca.core.tx.perf.Performance
      </programlisting>
      <para>
        class, where the following settings can be altered
        <itemizedlist>
          <listitem>
            CLIENTS: The number of clients in each run
          </listitem>
          <listitem>
            POOL_SIZES: The pool size in each run
          </listitem>
          <listitem>
            DO_RAMP_UP: Should ramp up be performed
          </listitem>
          <listitem>
            RAMP_UP_ITERATIONS: The number of iterations that the ramp-up should perform
          </listitem>
          <listitem>
            TRANSACTIONS_PER_CLIENT: The number of transactions each client should perform
          </listitem>
          <listitem>
            STATISTICS: Should statistics be enabled
          </listitem>
          <listitem>
            RECORD_ENLISTMENT_TRACES: Should enlistment be recorded
          </listitem>
          <listitem>
            USE_TRANSACTION_FOR_NOTRANSACTION: Use a transaction for NoTransaction scenario
          </listitem>
          <listitem>
            USE_CCM: Register connections with the cached connection manager
          </listitem>
          <listitem>
            TX_BEGIN_DURATION: The number of milliseconds spent in the resource during tx.begin()
          </listitem>
          <listitem>
            TX_COMMIT_DURATION: The number of milliseconds spent in the resource during tx.commit()
          </listitem>
          <listitem>
            MCP_IMPL: The implementation class of ManagedConnectionPool
          </listitem>
        </itemizedlist>
      </para>

      <para>
        A report can be generated using
      </para>
      <programlisting>
org.jboss.jca.core.tx.perf.PerfReport
      </programlisting>
      <para>
        which takes 3 arguments; output from NoopTS run, output from Narayana/MEM run and
        Narayana/FILE run.
      </para>

      <para>
        The data is presented on the console, and a GNU plot script is generated.
      </para>

      <para>
        The GNU plot can be generated using
      </para>
      <programlisting>
gnuplot perf.plot
      </programlisting>
      <para>
        which will generate a <code>perf.png</code> file with the graphs.
      </para>

      <para>
        Performance reports can be averaged using
      </para>
      <programlisting>
org.jboss.jca.core.tx.perf.AvgReport
      </programlisting>
      <para>
        which takes the .dat files from the <code>PerfReport</code> applications and generates
        a <code>perf-avg.dat</code> and a <code>perf-avg.plot</code> file.
      </para>

      <para>
        There is integration with JProfiler through the
      </para>
      <programlisting>
ant jprofiler
      </programlisting>
      <para>
        task. It is required to define the installation directory and the session
        id before the task is executed.
      </para>

      <para>
        The Bash scripts <code>perf-jprofiler.sh</code>, <code>perf-flightrecorder.sh</code>
        and <code>perf-perf.sh</code> are located in <code>core/src/test/resource</code> and
        can be used as a template for command line based runs, like
      </para>
      <programlisting>
ant prepare-test        
cp core/src/test/resource/perf-logging.properties build/core/test/logging.properties
cp core/src/test/resource/perf-perf.sh .
./perf-perf.sh
      </programlisting>
      <para>
        as an example for running the Linux perf script.
      </para>

    </section>

  </section>

</chapter>
