README Benchmark Tests
======================

This plugin provides automated benchmark tests and related support
programs to locate regressions or improvements between different versions
of the CVS Team Provider.

For now, all of the tests are targeted towards typical UI workflows.
This need not be the case.  Should it be necessary, benchmark tests can
easily be written for lower level core components as well.


Deploying the Tests
===================

1. If you have a pre-built plugin available, then you do not need to rebuild
   the org.eclipse.team.* plugins.  Otherwise you must build these as usual,
   but you do not need to create a JAR file.  Having all of the compiled
   classes in the 'bin' directory of each plugin is sufficient.

2. Checkout the matching version of the org.eclipse.team.tests.cvs.core source
   for your org.eclipse.team.* plugins.  Ensure all dependencies have been
   satisfied.  Compile as usual.
   
3. Checkout and compile org.eclipse.core.tests.harness.
   
4. On the target machine, install a matching version of Eclipse in a dedicated
   test directory.  Also install a suitable JRE for the platform into that
   directory (should always choose the same one).  

   If you are using Windows, then copy the "teamui_benchmark.bat" script to the
   directory.  For other platforms, you will need to cook up your own script.
   You will probably want to change the definition of the "ROOT" variable and
   other options before running the tests.
   
   Copy the template "repository.properties" file to the test directory.  Fill it in.
   
   Replace the existing org.eclipse.team.* plugins in the eclipse/plugins
   directory with the new ones.  Also install org.eclipse.core.tests.harness
   there.  In the typical case, this just means copying or exporting the projects
   directly from the workspace where you compiled them to the new location.
   
   The test directory structure should look like this:
     + mytestdirectory/
       - teamui_benchmark.bat
       - repository.properties
         + jre/
           ... a suitable java runtime ...
         + eclipse/
           - install/
           - readme/
           - splash/
           - buildmanifest.properties
           - startup.jar
           + plugins/
             - org.eclipse.core.tests.harness/
             - org.eclipse.team.core/
             - org.eclipse.team.ui/
             - org.eclipse.team.cvs.core/
             - org.eclipse.team.cvs.ui/
             - org.eclipse.team.tests.core/
             - org.eclipse.team.tests.cvs.core/
             - org.junit/
             ... and all of the other required Eclipse plugins ...
            
5. From a command shell, run the script.  Grab a coffee.


Running or Debugging the Tests from within Eclipse
==================================================

1. Checkout and compile the necessary projects (see above).

2. Using the PDE launcher, run the "org.eclipse.team.tests.cvs.core.harness"
   application with the following arguments:
   
   VM Arguments:
     -Declipse.cvs.properties=<location of your repository.properties file>
     
   Program arguments:
     -test <suite>   : id of suite to run (must be plugged into extension point)
                       [see plugin.xml file for the list of available tests]
     -log <file>     : specify a file for logging
     -nolog          : do not write a log file
     -repeat <n>     : number of iterations to run
     -ignorefirst    : ignore (do not record) results from first iteration
     -purge          : purge all projects from the workspace before each iteration
     <anything else> : passed verbatim to the org.eclipse.ui.workbench application


Inspecting the Output
=====================

1. Checkout and compile the org.eclipse.team.tests.cvs.core project.

2. Note that the log formatting tools require org.apache.xerces to be on the
   classpath when they are run.  They do not require any other Eclipse
   components, however.
   
3. Run any of the following Java programs:

   org.eclipse.team.tests.ccvs.ui.logformatter.PrintSummaryMain
   ------------------------------------------------------------
   
   Synopsis:
     Prints a summary of the output of all runs contained in a particular
     XML log file.  It is not possible to average runs in multiple log files
     at once without merging the files together on disk.  [Strip the closing
     tag of the first file, and the opening tag of the second file, then
     append the second file to the first]
   
   Program arguments:
     <log>       : the path of the log file to print
     -out <file> : specify an output file, otherwise sends to stdout
     -csv        : writes the data in comma separated values format
     -raw        : prints a raw dump without collating or averaging results


   org.eclipse.team.tests.ccvs.ui.logformatter.PrintDiffMain
   ---------------------------------------------------------

   Synopsis:
     Prints the difference between the average of all runs contained
     in one XML log file (the newer one) and the average of all runs
     contained in another XML log file (the older one).  This makes it
     possible to locate regressions or improvements between versions.
     
   Program arguments:
     <newer log> : the path of the "newer" log file
     <older log> : the path of the "older" log file
     -out <file> : specify an output file, otherwise sends to stdout
     -csv        : writes the data in comma separated values format
     -t <thresh> : specify the minimum non-negligible absolute % change
     -i          : ignore negligible changes in results [filter them out]


   org.eclipse.team.tests.ccvs.ui.logformatter.LogFormatterUIMain
   ---------------------------------------------------------

   Synopsis:
     Provides a GUI frontend to the log formatting tools.  To run this,
     you must ensure that the SWT DLL is on the java library path.


What is Being Logged
====================

At the present date the following information is logged for each test run:
  - current time
  - current SDK build number
  - for JUnit test cases:
    - test name
    - fully qualified class name
  - for groups of benchmark tasks:
    - name
  - for benchmark tasks:
    - name
    - elapsed time
  - for exceptions and errors:
    - type of error (warning, error, failure)
    - error message
    - stack trace, if applicable
    - printout of IStatus contents, if applicable


NOTES
=====

Exceptions and errors are not reported through the log formatting tools yet
since it is difficult to determine automatically which benchmark tasks are
affected (directly or indirectly) by the error.  For this reason, you should
MANUALLY INSPECT the generated XML log files and search for elements with
the name "abort".  Since running the test cases is time consuming, it
may be better to trim out any affected cases from the log rather than to run
the whole suite over once again.