emf/docs/performance/EMFPerformanceTestsFAQ.html - www.eclipse.org/modeling - Git at Google

 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
 <title>EMF Performance</title>
 </head>

 <body lang="EN-US" xml:lang="EN-US">

 <h1>EMF Performance Tests FAQ</h1>

 <p>
 Last updated: July 6, 2005
 </p>

 <p><b>What sort of dimension of performance is measured?</b></p>
 <blockquote>
 <p>CPU time is measured by the Eclipse performance suite.</p>
 </blockquote>

 <p><b>What setup (OS, Eclipse, JRE) was used to run the EMF
 performance suite?</b></p>

 <blockquote>
 <p><b>EMF:</b></p>
 <p>The base build is EMF 2.0.1 Release build:</p>
 <p><a
 href="http://download.eclipse.org/tools/emf/downloads/drops/2.0.1/R200409171617/">http://download.eclipse.org/tools/emf/downloads/drops/2.0.1/R200409171617/</a></p>
 <p>The compared build is EMF 2.1.0 RC1:</p>
 <p><a
 href="http://archive.eclipse.org/tools/emf/scripts/downloads.php#goto2.1.0S">http://archive.eclipse.org/tools/emf/scripts/downloads.php#goto2.1.0S</a></p>
 <p>Other than EMF, the other components are the
 same:</p>
 <p><b>Eclipse:</b></p>
 <p>The Eclipse build used is Eclipse 3.1RC2.</p>
 <p>Early trials indicated that using the same Eclipse driver that was used to build the EMF driver (eg., 2.0.1 with 3.0.1 vs. 2.1 with 3.1) introduced extra platform-specific changes. In order to minimize these platform-based effects, tests were performed using the same Eclipse version in all tests, even if that build did not exist at the time the driver was published or a newer driver was available.</p>

 <p><b>Java:</b></p>

 <p>The JRE used was IBM j9 (J2RE 1.4.2 IBM build
 cxia32142sr1a-20050209), with a heap size of 512MB, and J9 enabled.</p>
 <p><b>Ant:</b></p>
 <p>Apache Ant version 1.6.2 (Sept 28 2004) was used in conjunction with bash shell scripts to run the tests. These in turn then launched
 headless Eclipse to calculate the performance of the tests.</p>
 <p><b>Operating System:</b></p>

 <p>A Debian-based OS was chosen to run the tests due to its
 simplicity in setup. The particular variant was SimplyMEPIS 3.3, based on Debian
 GNU/Linux Unstable/Testing, running kernel 2.6.10. Very little configuration
 was done above the default MEPIS install, in order to minimize the work
 required to reproduce the configuration. </p>
 <p><b>Hardware:</b></p>

 <p>The machine used to run the tests is an IBM ThinkCentre PC
 with an <span style='color:black'>Intel(R) Pentium(R) 4 3.00GHz CPU </span>and
 1.5 GB memory.</p>
 </blockquote>

 <p><b>What is the EMF performance suite? Where do results come
 from?</b></p>
 <blockquote>
 <p>The major characteristic of the EMF performance suite is
 that most of the APIs tested have CPU times in the order of the micro-second or
 less. It should be seen as a micro-performance suite.</p>
 <p>The EMF performance suite uses an extension of the Eclipse
 performance suite (org.eclipse.test.performance), where a method is run several
 times (repetitions). The main difference is that the EMF performance suite uses
 the minimum instead of the average of all repetitions as the result of a test. To
 get the CPU time of a given method, that method is run for a number of
 repetitions, and then the result is the minimum of all the CPU times from all
 the repetitions. Therefore, the published results should be seen as the best
 possible time it can take to perform the operation. We wanted to have stable
 and reproducible results and we realized using the minimum provided the best
 measure of stability, as it excluded duration spikes and CPU swapping issues
 which contributed to both less stable data and wider ranging results.</p>
 <p>Because the APIs tested were faster than a millisecond,
 iterations were introduced (between when the measurements start and stop) to
 get results generally in the order of a few thousand milliseconds. We realized that
 increasing iterations was generally helping stability as well. The result of a
 single API call is computed from the result of the performance test divided by
 the number of iterations.</p>
 <p>The EMF performance tests suite is packaged as the
 org.eclipse.emf.test.performance plugin. You can get the EMF extension of the
 Eclipse performance suite from CVS here:</p>
 <p><a
 href="http://dev.eclipse.org/viewcvs/indextools.cgi/~checkout~/org.eclipse.emf.releng.build/tests/scripts/testperformance_emf.jar">http://dev.eclipse.org/viewcvs/indextools.cgi/~checkout~/org.eclipse.emf.releng.build/tests/scripts/testperformance_emf.jar</a></p>
 <p>For each build (e.g. EMF release 2.0.1), the EMF performance
 suite was run 10 times, to produce 10 minimum CPU times for each of the test
 methods. Minimum values were found based on a set of repetitions of a given
 test, discarding all but the smallest value.</p>
 <p>At most one of the 10 data points may have been deleted if
 it was much greater than the other 9 points. Then, average and standard
 deviation were computed on the reduced set of CPU times.</p>
 </blockquote>

 <p><b>How were improvements found?</b></p>
 <blockquote>
 <p>A test method is considered to have improved if, over 10 EMF
 performance suite runs, its average 2.1.0 CPU time is at least 5% less than its
 average 2.0.1 CPU time, and (2.1.0 average + 2.1.0 standard deviation) is less than
 (2.0.1 average - 2.0.1 standard deviation). Graphically, this means that the
 2.0.1 and 2.1.0 average times and their error bars do not overlap on a scatter
 plot of the values.</p>
 </blockquote>

 <p><b>What is each method doing? Where is the source code?</b></p>
 <blockquote>
 <p>You can browse the javadoc of the org.eclipse.emf.test.performance
 plugin here:</p>
 <p><a
 href="http://www.eclipse.org/emf/docs/performance/javadoc/">http://www.eclipse.org/emf/docs/performance/javadoc/</a></p>
 <p>You can browse the source of the org.eclipse.emf.test.performance
 plugin here:</p>
 <p><a
 href="http://dev.eclipse.org/viewcvs/indextools.cgi/org.eclipse.emf/tests/org.eclipse.emf.test.performance/src/org/eclipse/emf/test/performance/">http://dev.eclipse.org/viewcvs/indextools.cgi/org.eclipse.emf/tests/org.eclipse.emf.test.performance/src/org/eclipse/emf/test/performance/</a></p>
 <p>You can also extract the plugin from CVS. For details, see:</p>
 <p><a
 href="http://www.eclipse.org/emf/docs/?doc=docs/cvsdoc/emfcvsug.html">http://www.eclipse.org/emf/docs/?doc=docs/cvsdoc/emfcvsug.html</a></p>
 <p>Generally, a test method is run twice: once with a static EMF or SDO model instance, and once with a dynamic EMF or SDO model instance. This is implemented by having a static class extending the dynamic class with the same name, and having the static class override model initialization to use static instead of dynamic model. For this reason, the source of a (static or dynamic) method generally resides in the dynamic class.</p>
 </blockquote>

 </body>
 </html>
	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	<head>
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
	<title>EMF Performance</title>
	</head>

	<body lang="EN-US" xml:lang="EN-US">

	<h1>EMF Performance Tests FAQ</h1>

	<p>
	Last updated: July 6, 2005
	</p>

	<p><b>What sort of dimension of performance is measured?</b></p>
	<blockquote>
	<p>CPU time is measured by the Eclipse performance suite.</p>
	</blockquote>

	<p><b>What setup (OS, Eclipse, JRE) was used to run the EMF
	performance suite?</b></p>

	<blockquote>
	<p><b>EMF:</b></p>
	<p>The base build is EMF 2.0.1 Release build:</p>
	<p><a
	href="http://download.eclipse.org/tools/emf/downloads/drops/2.0.1/R200409171617/">http://download.eclipse.org/tools/emf/downloads/drops/2.0.1/R200409171617/</a></p>
	<p>The compared build is EMF 2.1.0 RC1:</p>
	<p><a
	href="http://archive.eclipse.org/tools/emf/scripts/downloads.php#goto2.1.0S">http://archive.eclipse.org/tools/emf/scripts/downloads.php#goto2.1.0S</a></p>
	<p>Other than EMF, the other components are the
	same:</p>
	<p><b>Eclipse:</b></p>
	<p>The Eclipse build used is Eclipse 3.1RC2.</p>
	<p>Early trials indicated that using the same Eclipse driver that was used to build the EMF driver (eg., 2.0.1 with 3.0.1 vs. 2.1 with 3.1) introduced extra platform-specific changes. In order to minimize these platform-based effects, tests were performed using the same Eclipse version in all tests, even if that build did not exist at the time the driver was published or a newer driver was available.</p>

	<p><b>Java:</b></p>

	<p>The JRE used was IBM j9 (J2RE 1.4.2 IBM build
	cxia32142sr1a-20050209), with a heap size of 512MB, and J9 enabled.</p>
	<p><b>Ant:</b></p>
	<p>Apache Ant version 1.6.2 (Sept 28 2004) was used in conjunction with bash shell scripts to run the tests. These in turn then launched
	headless Eclipse to calculate the performance of the tests.</p>
	<p><b>Operating System:</b></p>

	<p>A Debian-based OS was chosen to run the tests due to its
	simplicity in setup. The particular variant was SimplyMEPIS 3.3, based on Debian
	GNU/Linux Unstable/Testing, running kernel 2.6.10. Very little configuration
	was done above the default MEPIS install, in order to minimize the work
	required to reproduce the configuration. </p>
	<p><b>Hardware:</b></p>

	<p>The machine used to run the tests is an IBM ThinkCentre PC
	with an <span style='color:black'>Intel(R) Pentium(R) 4 3.00GHz CPU </span>and
	1.5 GB memory.</p>
	</blockquote>

	<p><b>What is the EMF performance suite? Where do results come
	from?</b></p>
	<blockquote>
	<p>The major characteristic of the EMF performance suite is
	that most of the APIs tested have CPU times in the order of the micro-second or
	less. It should be seen as a micro-performance suite.</p>
	<p>The EMF performance suite uses an extension of the Eclipse
	performance suite (org.eclipse.test.performance), where a method is run several
	times (repetitions). The main difference is that the EMF performance suite uses
	the minimum instead of the average of all repetitions as the result of a test. To
	get the CPU time of a given method, that method is run for a number of
	repetitions, and then the result is the minimum of all the CPU times from all
	the repetitions. Therefore, the published results should be seen as the best
	possible time it can take to perform the operation. We wanted to have stable
	and reproducible results and we realized using the minimum provided the best
	measure of stability, as it excluded duration spikes and CPU swapping issues
	which contributed to both less stable data and wider ranging results.</p>
	<p>Because the APIs tested were faster than a millisecond,
	iterations were introduced (between when the measurements start and stop) to
	get results generally in the order of a few thousand milliseconds. We realized that
	increasing iterations was generally helping stability as well. The result of a
	single API call is computed from the result of the performance test divided by
	the number of iterations.</p>
	<p>The EMF performance tests suite is packaged as the
	org.eclipse.emf.test.performance plugin. You can get the EMF extension of the
	Eclipse performance suite from CVS here:</p>
	<p><a
	href="http://dev.eclipse.org/viewcvs/indextools.cgi/~checkout~/org.eclipse.emf.releng.build/tests/scripts/testperformance_emf.jar">http://dev.eclipse.org/viewcvs/indextools.cgi/~checkout~/org.eclipse.emf.releng.build/tests/scripts/testperformance_emf.jar</a></p>
	<p>For each build (e.g. EMF release 2.0.1), the EMF performance
	suite was run 10 times, to produce 10 minimum CPU times for each of the test
	methods. Minimum values were found based on a set of repetitions of a given
	test, discarding all but the smallest value.</p>
	<p>At most one of the 10 data points may have been deleted if
	it was much greater than the other 9 points. Then, average and standard
	deviation were computed on the reduced set of CPU times.</p>
	</blockquote>

	<p><b>How were improvements found?</b></p>
	<blockquote>
	<p>A test method is considered to have improved if, over 10 EMF
	performance suite runs, its average 2.1.0 CPU time is at least 5% less than its
	average 2.0.1 CPU time, and (2.1.0 average + 2.1.0 standard deviation) is less than
	(2.0.1 average - 2.0.1 standard deviation). Graphically, this means that the
	2.0.1 and 2.1.0 average times and their error bars do not overlap on a scatter
	plot of the values.</p>
	</blockquote>

	<p><b>What is each method doing? Where is the source code?</b></p>
	<blockquote>
	<p>You can browse the javadoc of the org.eclipse.emf.test.performance
	plugin here:</p>
	<p><a
	href="http://www.eclipse.org/emf/docs/performance/javadoc/">http://www.eclipse.org/emf/docs/performance/javadoc/</a></p>
	<p>You can browse the source of the org.eclipse.emf.test.performance
	plugin here:</p>
	<p><a
	href="http://dev.eclipse.org/viewcvs/indextools.cgi/org.eclipse.emf/tests/org.eclipse.emf.test.performance/src/org/eclipse/emf/test/performance/">http://dev.eclipse.org/viewcvs/indextools.cgi/org.eclipse.emf/tests/org.eclipse.emf.test.performance/src/org/eclipse/emf/test/performance/</a></p>
	<p>You can also extract the plugin from CVS. For details, see:</p>
	<p><a
	href="http://www.eclipse.org/emf/docs/?doc=docs/cvsdoc/emfcvsug.html">http://www.eclipse.org/emf/docs/?doc=docs/cvsdoc/emfcvsug.html</a></p>
	<p>Generally, a test method is run twice: once with a static EMF or SDO model instance, and once with a dynamic EMF or SDO model instance. This is implemented by having a static class extending the dynamic class with the same name, and having the static class override model initialization to use static instead of dynamic model. For this reason, the source of a (static or dynamic) method generally resides in the dynamic class.</p>
	</blockquote>

	</body>
	</html>