| <html> |
| <head> |
| <title>Eclipse Testing Instructions</title> |
| </head> |
| |
| <body> |
| <h1>The Eclipse Test Framework</h1><p>Last Modified: March. 8, 2007</p><p><font size="+1"><br> |
| Introduction</font></p><p>The testing framework is comprised of the org.eclipse.test plugin and the org.eclipse.ant.optional.junit |
| fragment. </p><p>These two projects are available from the dev.eclipse.org repository and are |
| included in the eclipse-test-framework-<buildId>.zip from the eclipse.org |
| downloads page.</p><p><font size="+1">Building and Installation</font></p><p>Since the org.eclipse.test plugin is stored in the repository in source-code |
| form, it needs to be compiled before it can be used. The org.eclipse.ant.optional.junit |
| fragment does not contain any source and can be used as is from the repository.</p><ol> |
| <li>Turn of automatic builds. Window->Preferences->Workbench and uncheck "Perform |
| build automatically on resource modification"</li><li>Load org.eclipse.test into your workspace.</li><li>Right-click on the org.eclipse.test project in either the Navigator or Packages |
| view. Select 'Rebuild Project' from the context menu. This will compile the |
| entire org.eclipse.test plugin.</li><li>Finally, copy the org.eclipse.test plugin into your target Eclipse.</li> |
| <li>The org.eclipse.ant.optional.junit fragment only needs to be present in |
| the environment of the Eclipse that is overseeing the test process. If you |
| are running the test script from within the Workbench, this means that the |
| fragments need to be present withinn your development Eclipse. If you are |
| running the tests from the command line, then the fragments will need to be |
| present in your target Eclipse.</li></ol> |
| <p><font size="+1">Setup</font></p> |
| <p>Follow the steps given above to build and install all of the neccessary plugins |
| and fragments. Please note that the current version of the test framework is |
| not compatible with the PDE notion of self-hosting. If you want to run the tests, |
| you will need to setup a full target Eclipse so that the testing framework can |
| detect everything that is needed.</p> |
| <p>If you are writing tests for one or more Eclipse plugins, then you should |
| create a separate plugin for tests. The test plugin will contain all of the |
| tests that will be run on your development plugins, as well as defining how |
| those tests get run.</p> |
| |
| <p>If you are not writing tests for an Eclipse plugin, then you should look into |
| using JUnit on its own to test your work. JUnit is designed for testing Java |
| code, and by default has no knowledge of Eclipse. There are separate mechanisms |
| for using JUnit on Java code in Eclipse. See the documentation provided here:</p> |
| <p> <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/jdt-ui-home/plugins/org.eclipse.jdt.junit/index.html"> |
| JUnit Support in Eclipse</a><br> |
| <a href="http://dev.eclipse.org:8080/help/content/help:/org.eclipse.jdt.doc.user/tasks/tasks-207.htm?resultof=%6a%75%6e%69%74%20&toc=%2Forg.eclipse.jdt.doc.user%2Ftoc.xml"> |
| Using JUnit</a></p> |
| |
| <p>Once a test plugin has been created, you must then create an Ant script that |
| will run all of the tests. Create a file called 'test.xml' in the root of your plug-in or bundle. |
| This is an Ant file that defines how each test is going to be run. The 'test.xml' |
| file is like a DOS batch file that scripts the entire testing process. Typically, |
| the 'test.xml' file will contain targets for setting up the test run, executing |
| the tests, cleaning up afterwards, and running the entire process.</p> |
| |
| <p><font size="+1">Converting existing Test Suites</font></p> |
| |
| <p>If you are converting an existing set of tests to use the new framework, the |
| actual tests that have been written should not need much change.</p> |
| <p>If you have tests in multiple plugins, move these to a single test plug-in for your component.</p> |
| |
| <p>Make sure that the tests are defined in a plug-in. This is probably the most |
| common cause of confusion in the entire test process. Your tests need to be |
| in a plug-in so that Eclipse can find them when it tries to load them.</p> |
| |
| <p><font size="+1">Creating new Test Suites</font></p> |
| |
| <p>Creating new JUnit tests for an Eclipse plugin should be no more difficult |
| than writing standard JUnit tests. Since the framework allows tests to be run |
| inside of a working Eclipse, any tests that you write have available to them |
| any of the methods supplied by the Eclipse platform, provided that you add the |
| appropriate dependencies to your tests' manifest.</p> |
| |
| <p><font size="+1">Performance Issues</font></p> |
| <p>You should keep in mind the number of times that Eclipse needs to be |
| started. Launching Eclipse has a substantial cost in terms of runtime. To |
| minimize the number of times the platform is started, you should consider |
| writing a TestSuite called AllTests for each of your test plugins. AllTests |
| should invoke each of the tests that you want to run for a particular plugin. |
| The 'test.xml' file can then run the AllTests class, which will run all of your |
| tests, but the platform will only ever be started once for each of your test |
| plugins.</p><p>Note: Sometimes tests involve shutting down, restarting, and testing the state |
| of metadata that was written to disk. These session tests will require Eclipse |
| to be launched several times in sequence, which will increase the runtime of |
| your tests. This cannot be avoided.</p><p> </p> |
| |
| <p><font size="+1">Running the Test Suite from the UI</font></p><p>Right click on the test.xml file and select 'Run Ant...' from the pull-down |
| menu. The Running Ant dialog box pops up. From the list of targets, select the |
| one that runs all of your tests. If you are using the example file provided |
| below, this target is called 'Run', and will be selected by default. Hit the |
| 'Finish' button to start the test process.</p> |
| |
| <p><font size="+1">Running the Test Suite from the command line</font></p> |
| <p>When the test suites are invoked automatically, they are run from command |
| line. From the ${eclipse-home} directory, the command to use is:</p> |
| <code>java -jar plugins\org.eclipse.equinox.launcher_<version>.jar -application org.eclipse.ant.core.antRunner -buildfile ${test-plugin-path}\test.xml |
| -Declipse-home=${eclipse-home} -Dos=<operating system> -Dws=<windowing system> -Darch=<architecture></code> |
| |
| <p>Individual tests can also be invoked directly. From the ${eclipse-home} |
| directory, use the command:</p><code>java -jar plugins\org.eclipse.equinox.launcher_<version>.jar |
| -application ${launcher} -os <operating system> -ws <windowing system> -arch <architecture> -dev bin -testpluginname ${plugin-name} |
| -classname ${test-classname} formattter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,<path to output file with .xml extension></code> |
| <p>where ${launcher} is one of: <i>org.eclipse.test.uitestapplication</i> or <i>org.eclipse.test.coretestapplication</i> |
| depending on whether or not you want to run your tests within an active Workbench.</p><p><font size="+1">Output</font></p><p>By default, output from each test run is logged as XML. For each test that |
| is run, the output is logged into the file called ${classname}.xml. The individual |
| XML files are located in the ${eclipse-home} directory. When the test run is |
| finished, you can call the <b>"collect"</b> target in the library |
| file, which will collect the XML into a single file. See below for an example |
| of how to use this target correctly.</p> |
| |
| <h3>Other issues</h3><h4>Headless Testing vs. UI testing</h4><p>Many plugin tests will not need the Workbench active in order to run. Indeed, |
| only the minimum number of plugins needed to run the plugin being tested need |
| to be present when testing in a target Eclipse. There are two different Ant |
| targets provided for running Eclipse plugin tests. One target starts the entire |
| Workbench. The other starts Eclipse with the minimum number of plugins needed. |
| It is up to you to decide which target is most appropriate. For examples, look |
| at the <b>"ui-test"</b> and <b>"core-test"</b> targets below.</p> |
| |
| <h4>Disposing the Display</h4><p>Some low-level tests for the Eclipse platform take actions that are not |
| normally possible inside of Eclipse. An example of this behaviour would be |
| disposing the display. While this action can be performed while running the UI, |
| it will also kill the UI for the copy of Eclipse that is running, and cause |
| errors when the Workbench tries to shutdown. If you need to test disposing the |
| display, or other similar actions, your tests should be running without a UI. </p> |
| |
| <h4>Tests that are not plugins</h4><p>It is very easy to forget to define your tests inside of a plugin. If your |
| tests will not load properly, make sure that a plug-in manifest exists in your |
| test project, and also that the plugin is being loaded by the platform. Make |
| sure that all of the dependencies are satisfied for your test plugin.</p> |
| |
| <p><font size="+1">Interface</font></p><p>The org.eclipse.test plugin defines many useful Ant tasks/targets to aid developers |
| in writing their test.xml scripts. Currently, there is only Ant targets defined, |
| which can be called using Ant's built-in <ant> task. To use these targets, |
| add the following line to the top of your script, and reference the ${library-file} |
| property when calling <ant>:</p><code><property name="library-file" |
| value="${eclipse-home}/fragments/org.eclipse.test/library.xml"/> </code> |
| |
| <p>The targets that are defined are: </p><ul> |
| <li><b>ui-test</b> - This target runs a JUnit test suite inside of an Eclipse |
| Workbench. This target is mainly for testing plugins that use the Eclipse |
| UI and JFace. The output of the test pass is automatically logged in an XML |
| file called ${classname}.xml. It takes four arguments: </li><ul> |
| <li><i>data-dir</i> - The |
| data directory of the Eclipse that gets run</li><li><i>plugin-name</i> - |
| The name of the plugin that the test suite is defined in</li><li><i>classname</i> - |
| The name of the class that the test suite is defined in</li><li><i>vmargs</i> - An |
| optional argument string to pass to the VM running the tests</li></ul><p>For example, the following code will run the test |
| org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a new Eclipse |
| workbench. It passes the string "-Dbaz=true" to the VM. The Eclipse |
| stores its metadata in the directory "data-folder". |
| |
| <code><pre> |
| <ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}"> |
| <property name="data-dir" value="data-folder"/> |
| <property name="plugin-name" value="org.eclipse.foo"/> |
| <property name="classname" value="org.eclipse.foo.bar.MyTest"/> |
| <property name="vmargs" value="-Dbaz=true"/> |
| </ant> |
| </pre></code> |
| |
| <li><b>core-test</b> - This target runs a JUnit test suite inside of an IPlatformRunnable. |
| This target is for testing plugins that use the Eclipse platform, but do not |
| require a UI to be running. The output of the test pass is automatically logged |
| in an XML file called ${classname}.xml. It takes four arguments: </li><ul> |
| <li><i>data-dir</i> - The |
| data directory of the Eclipse that gets run</li><li><i>plugin-name</i> - |
| The name of the plugin that the test suite is defined in</li><li><i>classname</i> - |
| The name of the class that the test suite is defined in</li><li><i>vmargs</i> - An |
| optional argument string to pass to the VM running the tests</li></ul><p>For example, the following code will run the test |
| org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a headless Eclipse. |
| It passes the string "-Dbaz=true" to the VM. The Eclipse stores its |
| metadata in the directory "data-folder". |
| |
| <code><pre> |
| <ant target="core-test" antfile="${library-file}" dir="${eclipse-home}"> |
| <property name="data-dir" value="data-folder"/> |
| <property name="plugin-name" value="org.eclipse.foo"/> |
| <property name="classname" value="org.eclipse.foo.bar.MyTest"/> |
| <property name="vmargs" value="-Dbaz=true"/> |
| </ant> |
| </pre></code> |
| |
| <li><b>collect</b> - This |
| target collects the XML files that are produced over the course of the |
| test script. It takes two arguments: </li><ul> |
| <li><i>includes</i> - A |
| pattern matching all XML files to be included in the test report. This |
| argument is typically "org*.xml"</li><li><i>output-file</i> - |
| The filename where the output of the test gets stored. For the automated |
| build process, this file should be ${pluginname}.xml, and be located in |
| the ECLIPSE_HOME directory.</li></ul><p>For example, the following code collects all of the files |
| matching the pattern "org*.xml" in the directory ${eclipse-home}, |
| into the file named "logfile.xml". |
| |
| <code><pre> |
| <ant target="collect" antfile="${library-file}" dir="${eclipse-home}"> |
| <property name="includes" value="org*.xml"/> |
| <property name="output-file" value="logfile.xml"/> |
| </ant> |
| </pre></code> |
| |
| <li><b>test-regressions</b> - This target takes two compound XML files (after |
| a collect - even if collect is only given 1 file), and produces a text file |
| that contains a list of all test status changes that occur between the two |
| files. It will report new tests added, old tests that weren't run, and tests |
| whose status changed (pass to fail, or fail to pass). This target takes three |
| arguments: </li><ul> |
| <li><i>oldfile</i> - the |
| baseline file to compare against</li><li><i>newfile</i> - the |
| file containing the results of the most recent test run</li><li><i>outfile</i> - the |
| file in which to place the results of the regression test</li></ul><p>For example, the following code compares the test runs that |
| were logged into the files "old.xml" and "new.xml", and |
| creates a file called "output.txt" that holds the differences between |
| the two runs. |
| |
| <code><pre> |
| <ant target="test-regressions" antfile="${library-file}" dir="${eclipse-home}"> |
| <property name="oldfile" value="old.xml"/> |
| <property name="newfile" value="new.xml"/> |
| <property name="outfile" value="output.txt"/> |
| </ant> |
| </pre></code> |
| |
| <li><b>notify</b> - This target |
| sends email notification of the status of the latest test run. Typically, |
| this sends the results of the latest regression test to a developer so |
| that new test failures can be detected quickly. This target takes two |
| parameters: </li><ul> |
| <li><i>infile</i> - the |
| name of the file containing the message to send</li><li><i>address</i> - the |
| email address to send the file to</li></ul><p>For example, the following code will mail the contents of |
| the file "output.txt" to the address "foo@bar.com". |
| |
| <code><pre> |
| <ant target="notify" antfile="${library-file}" dir="${eclipse-home}"> |
| <property name="infile" value="output.txt"/> |
| <property name="address" value="foo@bar.com"/> |
| </ant> |
| </pre></code> |
| </ul><h2>Examples:</h2><p>Included is the 'test.xml' file from the org.eclipse.ui.tests plugin. This |
| file controls all of the automated testing that is done for the org.eclipse.ui |
| plugin. It can be run from inside of Eclipse or from the command line. It is |
| intended to serve as a template file for testing any other plugin.</p><p>Notice that the structure of the file roughly mirrors that of a JUnit test. |
| Targets are defined for setting up the tests, defining what needs to be done, |
| cleaning up after the tests, and running everything in the right order.</p><font color="red">This example is out of date. |
| <code><pre> |
| <?xml version="1.0"?> |
| |
| <project name="testsuite" default="run" basedir="."> |
| <!-- The property ${eclipse-home} should be passed into this script --> |
| <!-- Set a meaningful default value for when it is not. --> |
| <property name="eclipse-home" value="${basedir}/../.."/> |
| |
| <!-- sets the properties eclipse-home, and library-file --> |
| <property name="plugin-name" value="org.eclipse.ui.tests"/> |
| <property name="library-file" |
| value="${eclipse-home}/fragments/org.eclipse.test/library.xml"/> |
| |
| <!-- This target holds all initialization code that needs to be done for --> |
| <!-- all tests that are to be run. Initialization for individual tests --> |
| <!-- should be done within the body of the suite target. --> |
| <target name="init"> |
| <tstamp/> |
| <delete> |
| <fileset dir="${eclipse-home}" includes="org*.xml"/> |
| </delete> |
| </target> |
| |
| <!-- This target defines the tests that need to be run. --> |
| <target name="suite"> |
| <property name="session-folder" |
| value="${eclipse-home}/ui_session_sniff_folder"/> |
| <delete dir="${session-folder}" quiet="true"/> |
| <ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}"> |
| <property name="data-dir" value="${session-folder}"/> |
| <property name="plugin-name" value="${plugin-name}"/> |
| <property name="classname" |
| value="org.eclipse.ui.tests.api.SessionCreateTest"/> |
| </ant> |
| <ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}"> |
| <property name="data-dir" value="${session-folder}"/> |
| <property name="plugin-name" value="${plugin-name}"/> |
| <property name="classname" |
| value="org.eclipse.ui.tests.api.SessionRestoreTest"/> |
| </ant> |
| |
| <property name="sniff-folder" |
| value="${eclipse-home}/ui_sniff_folder"/> |
| <delete dir="${sniff-folder}" quiet="true"/> |
| <ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}"> |
| <property name="data-dir" value="${sniff-folder}"/> |
| <property name="plugin-name" value="${plugin-name}"/> |
| <property name="classname" |
| value="org.eclipse.ui.tests.UiTestSuite"/> |
| </ant> |
| </target> |
| |
| <!-- This target holds code to cleanup the testing environment after --> |
| <!-- after all of the tests have been run. You can use this target to --> |
| <!-- delete temporary files that have been created. --> |
| <target name="cleanup"> |
| </target> |
| |
| <!-- This target runs the test suite. Any actions that need to happen --> |
| <!-- after all the tests have been run should go here. --> |
| <target name="run" depends="init,suite,cleanup"> |
| <ant target="collect" antfile="${library-file}" dir="${eclipse-home}"> |
| <property name="includes" value="org*.xml"/> |
| <property name="output-file" value="${plugin-name}.xml"/> |
| </ant> |
| </target> |
| |
| </project> |
| </pre></code> |
| </font> |
| <h2>Known Issues:</h2><h4>Problem 1: Issues with Ant 1.3</h4><p>Ant 1.3 has some issues when used with the <style> tag and absolute |
| paths. Also, any tests that use the <code>System.exit()</code> call will not |
| log their output properly when using Ant 1.3.</p><h4>Problem 2: ECLIPSE_HOME</h4><p>The test suites need to know where the root of the eclipse install is on the |
| file system (the ECLIPSE_HOME variable). However, this variable is only defined |
| in JDT. The ${eclipse-home} property can be set to a reasonable default inside |
| of the test.xml script. Then tests can be run from the standard Ant window, |
| without having to specify -Declipse-home=%ECLIPSE_HOME%. If a value for |
| ${eclipse-home} does get passed in, the default (specified in test.xml) gets |
| overridden. The parameter is passed in by the build mechanism. For most cases, |
| the value "${basedir}/../.." is a reasonable default.</p><h4>Problem 3: Ugly reference to library.xml</h4><p>org.eclipse.test should provide Ant tasks, not template scripts.</p><h4>Problem 4: No console output</h4><p>When you run a TestSuite using the standard JUnit, it normally outputs a |
| series of dots to the console so that you can track the TestSuite's progress. |
| It is not possible to add this feature to the automated testing process at this |
| point in time.</p><h4>Problem 5: Ant java task on Linux</h4><p>Ant expects there to be a java executable on the system path. Furthermore, the executable |
| must be a real file, not a symbolic link. If the test framework is throwing an exception |
| <code>java.io.IOException: java: not found</code>, ensure that the java executable is on |
| your system path.</p><h4>Problem 6: PDE</h4><p>The testing framework currently has no knowledge of PDE. In order to run the automated |
| you must be running a self hosting environment with a full development and target Eclipse.</p> |
| |
| </body> |
| |
| </html> |
