development/tests.html - www.eclipse.org/webtools - Git at Google

 <!doctype html public "-//w3c//dtd html 4.0 transitional//en">
 <html>
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
 <link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
 <title>WTP automated testing</title>
 </head>

 <body>
 <table BORDER=0 CELLSPACING=5 CELLPADDING=2 WIDTH="100%" >
  <tr>
   <td ALIGN=LEFT width="80%">
    <p><b><font face="Verdana" size="+3">WTP automated testing</font></b><br>
    Last Updated: Nov. 29, 2004</p>
    <p>This document outlines the principles of WTP automated testing. It also provides some guidelines
    for creating and running test cases. If anyone has suggestions for this document,
    please post a message to the wtp-dev mailing list.</p>
   </td>
  </tr>
 </table>

 <table border=0 cellspacing=5 cellpadding=2 width="100%" >
  <tr>
   <td align=LEFT valign=TOP colspan="3" bgcolor="#0080C0"> <b><font face="Arial,Helvetica" color="#FFFFFF">Testing</font></b></td>
  </tr>
  <TR>
   <TD align="LEFT" valign="TOP">
   <p>The goal of unit testing is to achieve:
   <ol>
    <li>Continuous integration. Unit tests are run as part of the WTP builds. They give an early indication on
    what is failing. Unit test failures should be fixed as soon as possible (before the next integration build).</li>
    <li>API compatibility. Component teams should provide unit tests for their public APIs. This ensures developers
    do not break existing clients.</li>
   </ol>
   </p>
    <P>The goal of performance testing is to achieve uniform or better performance as WTP moves
    forwards. As a developer, your dedication to WTP performance is strongly desired. To ensure the
    performance of WTP does not regress over time, developers should provide performance test cases
    along side with their features, etc. Developers are also expected to verify their
    bug fixes and features contribution against existing performance test cases. If something is not performing
    well, open a bug. Use performance as the keyword. Click <a href="https://bugs.eclipse.org/bugs/buglist.cgi?short_desc_type=allwordssubstr&short_desc=&product=Web+Tools&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&keywords_type=allwords&keywords=performance&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailtype1=substring&email1=&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&changedin=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0=">here</a> to see a list of all the
    currently open performance bugs in WTP.</P>
    <p>Eclipse has a <a href="http://www.eclipse.org/test-and-performance/index.html">performance infrastructure</a> in place for measuring and tracking
    performance. The performance processes described in this document are modeled around the same
    infrastructure. To create and run tests under this infrastructure, please refer to the
    <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html">Eclipse Tests How-To document</a>.
    Eclipse also has tips and tools to aid developers to debug
    and track down performance problems. They are listed here:
    <ol>
     <li><a href="http://www.eclipse.org/eclipse/development/performance/bloopers.html">Performance bloopers</a></li>
     <li><a href="http://dev.eclipse.org/viewcvs/index.cgi/~checkout~/platform-core-home/dev.html#tools">Core tools</a></li>
     <li><a href="http://dev.eclipse.org/viewcvs/index.cgi/~checkout~/platform-swt-home/dev.html#tools">SWT tools</a></li>
    </ol>
    </p>
    <p>Components may have their own testing requirements. For example,
    the server tools framework often call APIs on server extensions to retrieve data about the
    extension. Some of these APIs must be short running as they are called from the UI. The server
    tools framework provides abstract performance test cases that extensions should extend to verify
    that code contributed by the extension does not regress performance in the base framework.
    Performance requirements from component teams are listed in a document which is located in that
    components development directory in CVS. Please refer to the <a href="http://www.eclipse.org/webtools/development/WTPDevelopmentPractice.html">WTP Development Practices</a>
    document regarding any development related issues.
    </p>
   </TD>
  </TR>
 </table>

 <table border=0 cellspacing=5 cellpadding=2 width="100%" >
  <tr>
   <td align=LEFT valign=TOP colspan="3" bgcolor="#0080C0"> <b><font face="Arial,Helvetica" color="#FFFFFF"><a name="policy"></a>Creating junit test cases</font></b></td>
  </tr>
  <tr>
   <td align="LEFT" valign="TOP">
    <p>Here's a laundry list for integrating test plugins into the WTP build:
    <ol>
     <li>Commit the plugin into CVS, use the component's "performance-tests" folder. For example,
     the org.eclipse.wst.wsdl.tests.performance plugin should be placed into
     /home/webtools/wst/components/wsdl/performance-tests.</li>
     <li>Add the plugin to the component's tests map file. This map file can be found inside
     the /home/webtools/org.eclipse.wtp.releng/maps directory.</li>
     <li>Add the plugin to the feature.xml file. This file can be found inside the
     /home/webtools/&lt;sub-project&gt;/assembly/features/&lt;performance feature&gt; directory.</li>
     <li>Update test.properties and test.xml inside /home/webtools/org.eclipse.wtp.releng/testScripts to include
     the new performance plugins.</li>
    </ol>
    The <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html">Eclipse Tests How-To document</a>
    has a very thorough explaination on how to create and run performance test cases using the Eclipse
    performance infrastructure.
    </p>
   </td>
  </tr>
 </table>

 <table border=0 cellspacing=5 cellpadding=2 width="100%" >
  <tr>
   <td align=LEFT valign=TOP colspan="3" bgcolor="#0080C0"> <b><font face="Arial,Helvetica" color="#FFFFFF">WTP performance process</font></b></td>
  </tr>
  <tr>
   <td align="LEFT" valign="TOP">
    <p>This section describes the process for tracking performance in WTP. It is based on the
    process used by the Eclipse Platform Project. All performance tests must be automated.
    Performance tests are run every week using Thursday's
    integration builds. Performance tests should:
    <ol>
     <li>never have compile errors</li>
     <li>always run to completion</li>
    </ol>
    </p>
    <p>If either condition fails, failures should be handled immediately according to the
    <a href="http://www.eclipse.org/webtools/development/WTPDevelopmentPractice.html">WTP Development Practices document</a>.
    Performance results are store in a Cloudscape database and are compared against results from
    the previous release. In case of a regression, a note will be posted to the mailing list
    indicating the problem.
    <ol>
     <li>the developer who introduced the regression should fix the performance problem</li>
     <li>if the regression can be justified by a new feature, then the PMC must get involved
     and decide how important that feature is (ex. any competition, etc). Solutions may
     include, but not limited to, making the feature optional (ex. create a preference and turn
     it off by default), so only users who wish to use it have to pay for it.</li>
    </ol>
    </p>
    <p>Performance results from the weekly integration build are rendered into a graph, and is
    linked to on the build page. This graph provides a simple comparison between the integration
    build and the reference build.
    </p>
    <p>To run performance tests for a build that's available from the Eclipse download Web site:
    <ol>
     <li>Check out /home/webtools/org.eclipse.wtp.releng</li>
     <li>Change the properties files to fit your system (buildAll.properties, tests.properties, build.cfg and etc)</li>
     <li>Open a command prompt and navigate to the org.eclipse.wtp.releng directory</li>
     <li>Run the following command:
      <p>ant -f cruise.xml -DbuildType=<buildType> -DbuildId=<buildId> -Dtimestamp=<timestamp> performance</p>
      <p>For example:</p>
      <p>ant -f cruise.xml -DbuildType=N -DbuildId=N20041127 -Dtimestamp=200411271458 performance</p>
     </li>
    </ol>
    Running performance tests for a local build is similar. Go to your ${buildDirectory} directory and check the
    buildType, buildId and timestamp for your local build. Go through the same steps as above. If you have a Cloudscape
    database setup (refer to the <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html">Eclipse Tests How-To document</a>),
    then the performance results will be written to the ${testDir}/results directory, else the performance results
    will be displayed in the console.
    </p>
   </td>
  </tr>
 </table>

 </html>
	<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
	<html>
	<head>
	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
	<link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
	<title>WTP automated testing</title>
	</head>

	<body>
	<table BORDER=0 CELLSPACING=5 CELLPADDING=2 WIDTH="100%" >
	<tr>
	<td ALIGN=LEFT width="80%">
	<p><b><font face="Verdana" size="+3">WTP automated testing</font></b><br>
	Last Updated: Nov. 29, 2004</p>
	<p>This document outlines the principles of WTP automated testing. It also provides some guidelines
	for creating and running test cases. If anyone has suggestions for this document,
	please post a message to the wtp-dev mailing list.</p>
	</td>
	</tr>
	</table>

	<table border=0 cellspacing=5 cellpadding=2 width="100%" >
	<tr>
	<td align=LEFT valign=TOP colspan="3" bgcolor="#0080C0"> <b><font face="Arial,Helvetica" color="#FFFFFF">Testing</font></b></td>
	</tr>
	<TR>
	<TD align="LEFT" valign="TOP">
	<p>The goal of unit testing is to achieve:
	<ol>
	<li>Continuous integration. Unit tests are run as part of the WTP builds. They give an early indication on
	what is failing. Unit test failures should be fixed as soon as possible (before the next integration build).</li>
	<li>API compatibility. Component teams should provide unit tests for their public APIs. This ensures developers
	do not break existing clients.</li>
	</ol>
	</p>
	<P>The goal of performance testing is to achieve uniform or better performance as WTP moves
	forwards. As a developer, your dedication to WTP performance is strongly desired. To ensure the
	performance of WTP does not regress over time, developers should provide performance test cases
	along side with their features, etc. Developers are also expected to verify their
	bug fixes and features contribution against existing performance test cases. If something is not performing
	well, open a bug. Use performance as the keyword. Click <a href="https://bugs.eclipse.org/bugs/buglist.cgi?short_desc_type=allwordssubstr&short_desc=&product=Web+Tools&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&keywords_type=allwords&keywords=performance&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailtype1=substring&email1=&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&changedin=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0=">here</a> to see a list of all the
	currently open performance bugs in WTP.</P>
	<p>Eclipse has a <a href="http://www.eclipse.org/test-and-performance/index.html">performance infrastructure</a> in place for measuring and tracking
	performance. The performance processes described in this document are modeled around the same
	infrastructure. To create and run tests under this infrastructure, please refer to the
	<a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html">Eclipse Tests How-To document</a>.
	Eclipse also has tips and tools to aid developers to debug
	and track down performance problems. They are listed here:
	<ol>
	<li><a href="http://www.eclipse.org/eclipse/development/performance/bloopers.html">Performance bloopers</a></li>
	<li><a href="http://dev.eclipse.org/viewcvs/index.cgi/~checkout~/platform-core-home/dev.html#tools">Core tools</a></li>
	<li><a href="http://dev.eclipse.org/viewcvs/index.cgi/~checkout~/platform-swt-home/dev.html#tools">SWT tools</a></li>
	</ol>
	</p>
	<p>Components may have their own testing requirements. For example,
	the server tools framework often call APIs on server extensions to retrieve data about the
	extension. Some of these APIs must be short running as they are called from the UI. The server
	tools framework provides abstract performance test cases that extensions should extend to verify
	that code contributed by the extension does not regress performance in the base framework.
	Performance requirements from component teams are listed in a document which is located in that
	components development directory in CVS. Please refer to the <a href="http://www.eclipse.org/webtools/development/WTPDevelopmentPractice.html">WTP Development Practices</a>
	document regarding any development related issues.
	</p>
	</TD>
	</TR>
	</table>

	<table border=0 cellspacing=5 cellpadding=2 width="100%" >
	<tr>
	<td align=LEFT valign=TOP colspan="3" bgcolor="#0080C0"> <b><font face="Arial,Helvetica" color="#FFFFFF"><a name="policy"></a>Creating junit test cases</font></b></td>
	</tr>
	<tr>
	<td align="LEFT" valign="TOP">
	<p>Here's a laundry list for integrating test plugins into the WTP build:
	<ol>
	<li>Commit the plugin into CVS, use the component's "performance-tests" folder. For example,
	the org.eclipse.wst.wsdl.tests.performance plugin should be placed into
	/home/webtools/wst/components/wsdl/performance-tests.</li>
	<li>Add the plugin to the component's tests map file. This map file can be found inside
	the /home/webtools/org.eclipse.wtp.releng/maps directory.</li>
	<li>Add the plugin to the feature.xml file. This file can be found inside the
	/home/webtools/<sub-project>/assembly/features/<performance feature> directory.</li>
	<li>Update test.properties and test.xml inside /home/webtools/org.eclipse.wtp.releng/testScripts to include
	the new performance plugins.</li>
	</ol>
	The <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html">Eclipse Tests How-To document</a>
	has a very thorough explaination on how to create and run performance test cases using the Eclipse
	performance infrastructure.
	</p>
	</td>
	</tr>
	</table>

	<table border=0 cellspacing=5 cellpadding=2 width="100%" >
	<tr>
	<td align=LEFT valign=TOP colspan="3" bgcolor="#0080C0"> <b><font face="Arial,Helvetica" color="#FFFFFF">WTP performance process</font></b></td>
	</tr>
	<tr>
	<td align="LEFT" valign="TOP">
	<p>This section describes the process for tracking performance in WTP. It is based on the
	process used by the Eclipse Platform Project. All performance tests must be automated.
	Performance tests are run every week using Thursday's
	integration builds. Performance tests should:
	<ol>
	<li>never have compile errors</li>
	<li>always run to completion</li>
	</ol>
	</p>
	<p>If either condition fails, failures should be handled immediately according to the
	<a href="http://www.eclipse.org/webtools/development/WTPDevelopmentPractice.html">WTP Development Practices document</a>.
	Performance results are store in a Cloudscape database and are compared against results from
	the previous release. In case of a regression, a note will be posted to the mailing list
	indicating the problem.
	<ol>
	<li>the developer who introduced the regression should fix the performance problem</li>
	<li>if the regression can be justified by a new feature, then the PMC must get involved
	and decide how important that feature is (ex. any competition, etc). Solutions may
	include, but not limited to, making the feature optional (ex. create a preference and turn
	it off by default), so only users who wish to use it have to pay for it.</li>
	</ol>
	</p>
	<p>Performance results from the weekly integration build are rendered into a graph, and is
	linked to on the build page. This graph provides a simple comparison between the integration
	build and the reference build.
	</p>
	<p>To run performance tests for a build that's available from the Eclipse download Web site:
	<ol>
	<li>Check out /home/webtools/org.eclipse.wtp.releng</li>
	<li>Change the properties files to fit your system (buildAll.properties, tests.properties, build.cfg and etc)</li>
	<li>Open a command prompt and navigate to the org.eclipse.wtp.releng directory</li>
	<li>Run the following command:
	<p>ant -f cruise.xml -DbuildType=<buildType> -DbuildId=<buildId> -Dtimestamp=<timestamp> performance</p>
	<p>For example:</p>
	<p>ant -f cruise.xml -DbuildType=N -DbuildId=N20041127 -Dtimestamp=200411271458 performance</p>
	</li>
	</ol>
	Running performance tests for a local build is similar. Go to your ${buildDirectory} directory and check the
	buildType, buildId and timestamp for your local build. Go through the same steps as above. If you have a Cloudscape
	database setup (refer to the <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html">Eclipse Tests How-To document</a>),
	then the performance results will be written to the ${testDir}/results directory, else the performance results
	will be displayed in the console.
	</p>
	</td>
	</tr>
	</table>

	</html>