bundles/org.eclipse.test.performance.ui/html/preferences.html - platform/eclipse.platform.releng.buildtools - Git at Google

 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">

 <html>
 <head>
     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
     <title>Preferences</title>
 </head>

 <body>
 <h1>Preferences</h1>
 <p>
 The tool have some preferences which may be configured by advanced users.
 This must be done carefully otherwise, either no data could be read by the tool
 and/or generated results could be puzzled.
 </p><p>

 <h3><a name="eclipse_version">Eclipse version</a></h3>
 <p><img src="images/preferences-eclipse-versions.png" alt="Preferences Eclipse versions"/></p>
 <p>
 The Eclipse version on which the performance results belongs to. There are
 only two possible choice: the maintenance and the current versions.
 </p>

 <h3><a name="database">Database</a></h3>
 <p><img src="images/preferences-database.png" alt="Preferences Database"/></p>
 <p>
 By default the tool does not connect to any performance results database as common
 users might not have enough rights to access it. However, users having these
 rights may want to look at the database contents and update the local data files
 with it. Hence, it is possible to configure the tool to connect to a database
 which may be either local or on the releng server (<code><b>minsky</b></code>).
 </p><p>
 Note that the folder for the local database must be the parent of the
 <code>perfDb3x</code> folder otherwise you'll get an error message.
 </p>

 <h3><a name="status">Status</a></h3>
 <p><img src="images/preferences-status.png" alt="Preferences Status"/></p>
 <p>
 The status preferences can be set to exclude some tests while written
 the status file (see <a href="components.html#writestatus">Write status</a>).
 </p><p>
 Flying over each check-box or radio buttons gives a short explanation of each
 preference value
 </p><p>
 Here are the detailed explanation of these status preferences:
 </p>
 <ul>
 <li><b>Values</b>: Check-box to include the values of the failing tests in the
 status file. Note that is not recommended to do so when status want to be
 compared between builds</li>
 <li><b>Error level</b>: Level of the error from which the test result is
 excluded from the file:
 <ul>
 <li>None: the test is always included in the status file whatever the error
 level is</li>
 <li>Invalid: the test is not included when the error level is over 100%</li>
 <li>Weird: the test is not included when the error level is over 50%</li>
 <li>Suspicious: the test is not included when the error level is over 25%</li>
 <li>Noticeable: the test is not included when the error level is over 3%</li>
 </ul>
 </li>
 <li><b>Small value</b>: The test is not included when a small value is detected:
 <ul>
 <li>Build value: the test is not included when the value for this build test value
 is below than 100ms (<i>as the test duration is below the recommended minimum
 value it's not necessary to strongly survey it...</i>)</li>
 <li>Delta value: the test is not included when the value of the difference between
 the build and the baseline is below than 100ms (<i>as the regression is below
 what a normal user can detect, it may not be necessary to report it...</i>)</li>
 </ul>
 </li>
 <li><b>Statistics</b>: Level of deviation of test history since the first build
 from which the test result is not included in the status file:
 <ul>
 <li>None: the test is always included in the status file whatever the deviation
 is</li>
 <li>Unstable: the test is not included when the deviation is over 10%</li>
 <li>Erratic: the test is not included when the error level is over 20%</li>
 </ul>
 </li>
 <li><b>Builds to confirm</b>: The number of builds to confirm a regression.
 As tests may have <i>natural</i> variation, it's often necessary to have several
 builds to confirm that a regression really occurs. This preference allow to
 define how many consecutive builds must show a regression before including
 a test result in the status file...
 </li>
 </ul>

 <h3><a name="milestones">Milestones</a></h3>
 <p><img src="images/preferences-milestones.png" alt="Preferences Milestones"/></p>
 <p>
 These are the list of the version milestones. Each milestone is a date string
 using the <b>yyyymmddHHMM</b> format. When a new milestone is shipped, then a new
 date must be added to this preference to let the tool to identify it in the builds
 list and emphasize it...
 </p>

 <h3><a name="lastbuild">Last build</a></h3>
 <p><img src="images/preferences-lastbuild.png" alt="Preferences Last build"/></p>
 <p>
 The last build on which verifications and/or generation want to be done. When
 not set (which is the default value) the last build of the database is taken
 into account.
 </p><p>
 All builds after the selected one are ignored by the tool likewise they would
 have no local data. Changing this value will trigger the initialization of
 the local data which will be read again.
 </p>

 <h2><a name="defaultdim">Default dimension</a></h2>
 <p><img src="images/preferences-default-dim.png" alt="Preferences Default dimension"/></p>
 <p>
 This is the dimension used to compute delta and make the verification. Currently
 this is the <b>Elapsed Process Time</b> dimension.
 </p><p>
 <i>Note that the default dimension must belong to the <b>Results dimensions</b>
 described below, hence a new selected dimensions will always be automatically
 added to the list...</i>
 </p>

 <h2><a name="resultsdim">Results dimensions</a></h2>
 <p><img src="images/preferences-results-dim.png" alt="Preferences Results dimension"/></p>
 <p>
 These are dimensions displayed in the scenario data HTML pages. Currently there
 are the <b>Elapsed Process Time</b> and the <b>CPU Time</b>. Having these dimensions
 configurable may be interesting to display others dimensions and see whether their
 numbers may be relevant or not (e.g. <b>Used Hava Heap</b>).
 </p><p>
 <i>Note that the default dimension described above must belong to the selected
 dimensions, hence it will always be automatically added to the new selected list...</i>
 </p>
 </body>
 </html>
	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">

	<html>
	<head>
	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
	<title>Preferences</title>
	</head>

	<body>
	<h1>Preferences</h1>
	<p>
	The tool have some preferences which may be configured by advanced users.
	This must be done carefully otherwise, either no data could be read by the tool
	and/or generated results could be puzzled.
	</p><p>

	<h3><a name="eclipse_version">Eclipse version</a></h3>
	<p><img src="images/preferences-eclipse-versions.png" alt="Preferences Eclipse versions"/></p>
	<p>
	The Eclipse version on which the performance results belongs to. There are
	only two possible choice: the maintenance and the current versions.
	</p>

	<h3><a name="database">Database</a></h3>
	<p><img src="images/preferences-database.png" alt="Preferences Database"/></p>
	<p>
	By default the tool does not connect to any performance results database as common
	users might not have enough rights to access it. However, users having these
	rights may want to look at the database contents and update the local data files
	with it. Hence, it is possible to configure the tool to connect to a database
	which may be either local or on the releng server (<code><b>minsky</b></code>).
	</p><p>
	Note that the folder for the local database must be the parent of the
	<code>perfDb3x</code> folder otherwise you'll get an error message.
	</p>

	<h3><a name="status">Status</a></h3>
	<p><img src="images/preferences-status.png" alt="Preferences Status"/></p>
	<p>
	The status preferences can be set to exclude some tests while written
	the status file (see <a href="components.html#writestatus">Write status</a>).
	</p><p>
	Flying over each check-box or radio buttons gives a short explanation of each
	preference value
	</p><p>
	Here are the detailed explanation of these status preferences:
	</p>
	<ul>
	<li><b>Values</b>: Check-box to include the values of the failing tests in the
	status file. Note that is not recommended to do so when status want to be
	compared between builds</li>
	<li><b>Error level</b>: Level of the error from which the test result is
	excluded from the file:
	<ul>
	<li>None: the test is always included in the status file whatever the error
	level is</li>
	<li>Invalid: the test is not included when the error level is over 100%</li>
	<li>Weird: the test is not included when the error level is over 50%</li>
	<li>Suspicious: the test is not included when the error level is over 25%</li>
	<li>Noticeable: the test is not included when the error level is over 3%</li>
	</ul>
	</li>
	<li><b>Small value</b>: The test is not included when a small value is detected:
	<ul>
	<li>Build value: the test is not included when the value for this build test value
	is below than 100ms (<i>as the test duration is below the recommended minimum
	value it's not necessary to strongly survey it...</i>)</li>
	<li>Delta value: the test is not included when the value of the difference between
	the build and the baseline is below than 100ms (<i>as the regression is below
	what a normal user can detect, it may not be necessary to report it...</i>)</li>
	</ul>
	</li>
	<li><b>Statistics</b>: Level of deviation of test history since the first build
	from which the test result is not included in the status file:
	<ul>
	<li>None: the test is always included in the status file whatever the deviation
	is</li>
	<li>Unstable: the test is not included when the deviation is over 10%</li>
	<li>Erratic: the test is not included when the error level is over 20%</li>
	</ul>
	</li>
	<li><b>Builds to confirm</b>: The number of builds to confirm a regression.
	As tests may have <i>natural</i> variation, it's often necessary to have several
	builds to confirm that a regression really occurs. This preference allow to
	define how many consecutive builds must show a regression before including
	a test result in the status file...
	</li>
	</ul>

	<h3><a name="milestones">Milestones</a></h3>
	<p><img src="images/preferences-milestones.png" alt="Preferences Milestones"/></p>
	<p>
	These are the list of the version milestones. Each milestone is a date string
	using the <b>yyyymmddHHMM</b> format. When a new milestone is shipped, then a new
	date must be added to this preference to let the tool to identify it in the builds
	list and emphasize it...
	</p>

	<h3><a name="lastbuild">Last build</a></h3>
	<p><img src="images/preferences-lastbuild.png" alt="Preferences Last build"/></p>
	<p>
	The last build on which verifications and/or generation want to be done. When
	not set (which is the default value) the last build of the database is taken
	into account.
	</p><p>
	All builds after the selected one are ignored by the tool likewise they would
	have no local data. Changing this value will trigger the initialization of
	the local data which will be read again.
	</p>

	<h2><a name="defaultdim">Default dimension</a></h2>
	<p><img src="images/preferences-default-dim.png" alt="Preferences Default dimension"/></p>
	<p>
	This is the dimension used to compute delta and make the verification. Currently
	this is the <b>Elapsed Process Time</b> dimension.
	</p><p>
	<i>Note that the default dimension must belong to the <b>Results dimensions</b>
	described below, hence a new selected dimensions will always be automatically
	added to the list...</i>
	</p>

	<h2><a name="resultsdim">Results dimensions</a></h2>
	<p><img src="images/preferences-results-dim.png" alt="Preferences Results dimension"/></p>
	<p>
	These are dimensions displayed in the scenario data HTML pages. Currently there
	are the <b>Elapsed Process Time</b> and the <b>CPU Time</b>. Having these dimensions
	configurable may be interesting to display others dimensions and see whether their
	numbers may be relevant or not (e.g. <b>Used Hava Heap</b>).
	</p><p>
	<i>Note that the default dimension described above must belong to the selected
	dimensions, hence it will always be automatically added to the new selected list...</i>
	</p>
	</body>
	</html>