| <html> |
| |
| <head> |
| <meta http-equiv=Content-Type content="text/html; charset=windows-1252"> |
| <title>Using TPTP to Automate Functional Testing</title> |
| |
| <link rel=Stylesheet type="text/css" media=all |
| href="../Article-TPTPAutomateFunctionalTesting/default_style.css"> |
| <link rel="stylesheet" href="../article.css" type="text/css" /> |
| </head> |
| |
| <body> |
| <h1>Using TPTP to Automate Functional Testing</h1> |
| |
| <div class="summary"> |
| <h2>Summary</h2> |
| This article provides an overview and some details of the design of a |
| functional test automation solution that was built using testing |
| frameworks available within the TPTP Testing Tools Project. |
| |
| <div class="author">By Govin Varadarajan & Kim Letkeman, IBM |
| Canada</div> |
| <div class="copyright">Copyright © 2007 International |
| Business Machines Corp.</div> |
| <div class="date">March 29, 2007</div> |
| </div> |
| |
| <div class="content"> |
| <h1>Introduction</h1> |
| |
| <p>This paper discusses the design of a functional and performance |
| testing solution using TPTP [1] and TPTP Automated GUI Recorder [2]. |
| This solution is used within the Rational Modeling Platform groups Test |
| Automation system (RMPTAS) [5] to execute functional and performance |
| tests. The Test Automation System is an end-to-end testing framework, |
| supporting the execution of multiple types of tests and including a |
| front-end viewer (a web interface for displaying the test results) and a |
| back-end (a database to store results from test executions).</p> |
| |
| <h2><a name="_Toc154289790">Background and Philosophy</a></h2> |
| |
| <p>The Rational Modeling team has had automated tests for a long |
| time, but these tests required significant maintenance and their |
| framework required too much baby-sitting to really be called <i>automated.</i></p> |
| |
| <p>So the team began again from scratch with these basic guidelines:</p> |
| |
| <ul> |
| <li>The system must reliably see every relevant build and must |
| respond without manual intervention</li> |
| <li>The system must be easily maintained, which required that it |
| be a modular design and that it use well-supported components</li> |
| <li>Writing automated tests must be easy … that is, |
| unnecessary barriers to entry must be eliminated</li> |
| <li>There must be as little extra overhead for developers as |
| possible, i.e. no extra applications to install</li> |
| <li>Tests should be able to verify everything, not just GUI |
| results, so a white box flavor is desirable</li> |
| <li>It must be possible to validate test executions without the |
| use of technologies such as bitmap comparison that render the test |
| cases brittle.</li> |
| </ul> |
| |
| <p>The fundamental goal for the test automation solution is to |
| relieve developers of the burden of manual regression testing. |
| Regression testing is extremely repetitive, and developers find it |
| boring after the first time through. Running regression tests eventually |
| becomes a study in quick and painless execution, which of course means |
| that issues are missed.</p> |
| |
| <p>The result is false negatives, where a failure is missed simply |
| because the developer glazes over from too deep a familiarity with the |
| test cases. Ironically, false positives are also common, where failures |
| are caught in later regression tests, leading development to incorrectly |
| assume a recent break and to waste a lot of debugging time.</p> |
| |
| <h2><a name="_Toc154289791">Technology</a></h2> |
| |
| <p>For the automation framework, a custom perl-script-based solution |
| was rejected as it has proven unreliable in the past. Instead, the |
| Software Test Automation Framework (STAF), an open-source solution from |
| IBMs Automated Integration and Middleware division was chosen as a |
| well-supported automation infra-structure. </p> |
| |
| <p>For the test scripting language and supporting tools, several |
| approaches were examined. Abbot, a tool allowing automation of |
| keystrokes and mouse events, was explored in depth because of its |
| ability to be run in the developers environment with minimal overhead. |
| Tests can be crafted using the familiar JUnit style. Where Abbot came up |
| short was in support (none) and script recording (awkward using the |
| related Costello tools.)</p> |
| |
| |
| <p>External applications were also examined, and are in fact |
| supported in our automation framework. Rational Functional Tester (RFT) |
| is a powerful test application with a vibrant community and plenty of |
| support. However, in a developer's environment, the sheer size of the |
| installation is a barrier to entry; as well, RFT is designed to run |
| along side of the application under test, requiring very much a black |
| box approach. This is perfect for use by dedicated test teams, but is |
| not as efficient when used by development teams, as it makes it more |
| difficult to use white box techniques for verification points.</p> |
| |
| <p>The automation project was turned on its head with the arrival of |
| the Automated GUI Recorder (AGR) from the Test and Performance Tools |
| Platform (TPTP) open source project. This is a very easy-to-use test |
| case recording system that creates simple XML scripts into which |
| verification hooks can easily be inserted and tracked by the test |
| automation framework. Stylistically, it uses the very familiar JUnit |
| framework and is fully extensible. Work is underway to complete the |
| missing features (drag and drop and GEF figure and palette support) in |
| order to have a complete test automation solution that is not sensitive |
| to minor GUI changes for test case verification.</p> |
| |
| <p>It is important to clarify what minor GUI changes means in this |
| context. An example is a difference in the layout of a diagram between |
| two runs of a test case. When bitmap positions are remembered for the |
| entire diagram, even tiny changes in position can cause false positives. |
| During GUI development cycles, layout algorithms are subject to change, |
| which can render the location of an item on the diagram surface |
| differently by a few pixels in a subsequent execution. This makes any |
| bitmap comparison between the full diagrams brittle. Another source of |
| false differences is the slight change in colors that can occur |
| throughout the refinement stages of GUI development. Bitmap comparisons |
| are brittle by definition, and get more and more complex as attempts are |
| made to become color independent, and to create fuzzy location matching.</p> |
| |
| <p>An important aspect of an automated test solution is the ability |
| to verify test executions for correctness. In the earlier |
| perl-script-based system, verification was achieved using custom bitmap |
| comparison techniques, which were brittle and produced frequent false |
| positives. In the new solution we wanted to avoid the use of bitmap |
| comparisons, and instead use custom verification libraries that directly |
| examine (using object-based recognition) our data to confirm the |
| correctness of the previous step(s.) The AGRs verification support via |
| the <i>verification hook mechanism [3] </i>supports our approach because |
| it provides a convenient means for invoking custom verification |
| libraries that do not use any bitmap comparisons. Details on the |
| "verification hook" based verification mechanism provided by |
| AGR may be found in the section entitled "Test Verification" |
| later in this article and the AGR User Guide [3].</p> |
| |
| <p>Testers and developers can record AGR-based functional tests, |
| augmenting them with verification points that use well-tested libraries. |
| Included are methods for finding named elements in a model and the like. |
| Tests will be expected to follow naming and structure guidelines and to |
| be added into our configuration management system, where the automation |
| system will pick and execute them on every product build. Both component |
| assembly and packaged installation builds are supported. Great care |
| should be taken to avoid introducing unnecessary behavior difference in |
| your installs, so that your tests can move seamlessly between test |
| development and production environments.</p> |
| |
| <p><i>Note that, as of March 2007, a solution was found that |
| provides a stable and reliable method for recording and playing back |
| diagram and palette events. The basic method was to extend the event |
| stream to include mouse events from diagram and palette controls. The |
| mouse events are anchored to the smallest window possible, which removes |
| the brittleness of location as an issue (i.e. location of a figure on a |
| diagram is not relevant, only location of the mouse event within the |
| figure a much less brittle method.) The events are played back after |
| obtaining the figures edit part. This technique requires that a unique |
| identifier be generated for each figure and that the figure be retrieved |
| just before playing events destined for it. The details are too complex |
| for any further explanation and will require another article. The key |
| point is that we feel that the AGR solution is now complete to the level |
| needed to fully test our application.</i></p> |
| |
| <h1><a name="_Toc154289792">Testing Solution Design</a></h1> |
| |
| <p>Our functional test automation solution uses TPTP AGR 4.2.1 to |
| record and playback tests and TPTP Automatable Service Framework (ASF) |
| to automate the execution of tests and querying of test results. For |
| those who are not very familiar with the TPTP technologies that are used |
| in this solution, some back ground information is provided in Appendix |
| -A</p> |
| |
| <h2>Overview</h2> |
| |
| <p>Figure 1 shows a high level view of the solution design with TPTP |
| platform components in light green and our custom components in dark |
| green. As seen in Figure 1, the TPTP Testing Tools Project, which is |
| based on the TPTP Platform Project; provides means to create and execute |
| various types of tests such as JUnit, URL (Web stress testing), AGR and |
| Manual tests. TPTP models test suites containing these types of tests |
| in instances of EMF models implementing open-source test data model |
| specifications such as the OMGs UML2 Testing Profile (UML2TP). When |
| any of these types of tests are executed, test results become available |
| in execution history files which are also EMF model instances. These |
| execution history files may be viewed using the Test Log view, reported |
| using the integrated and extensible BIRT reports and queried for test |
| execution information and results. </p> |
| |
| <div class="figure"><img src="images/image001.jpg"><br> |
| Figure 1: Test Automation Solution Design</div> |
| |
| <p>For test automation, execution of tests may be automated using a |
| Test Execution Service that the TPTP Automatable Service Framework (ASF) |
| provides. This service may be invoked using a command-line |
| interface/adapter that the service implementation provides. In order to |
| completely automate a test solution, one also needs a mechanism to |
| automatically query execution history files. At the time of developing |
| this solution, no suitable facility was available from TPTP for querying |
| the execution history files. Hence, a new ASF service named Model Query |
| service was implemented to query execution history files.</p> |
| |
| <p>In our functional test automation solution design, AGR tests are |
| executed using the ASF Test Execution services command-line |
| interface. Artifacts produced during the execution of the tests are |
| semantically verified using a custom Model Verification Library, thereby |
| confirming the successful execution of tests. The execution history |
| files resulting from the test execution are queried for results using |
| the Model Query service that we implemented. This Model Query service |
| is invoked using a command-line interface/adapter that is provided by |
| the service implementation. The information obtained upon querying the |
| execution histories is written to a log file for further use, although |
| alternatively this information could be directly written to a database |
| using another ASF service (referred to as database mutation service, in |
| Figure 1). The RMPTAS reads the log file and populates a database with |
| the test results.</p> |
| |
| <p>While the above section provides an over view of the solution |
| design, the next few sections describe the various components of the |
| test automation solution design in some detail.</p> |
| |
| <h2><a name="_Toc154289793">Test Execution Service</a></h2> |
| |
| <p>As mentioned above, ASF provides a test execution service for the |
| programmatic launch, execution and generation of test results. With the |
| test execution service, tests can be launched using a headless Eclipse |
| from ant scripts, shell scripts and arbitrary programs. Multiple tests |
| can be run at the same time given enough memory on the executing machine |
| (each service execution currently requires a separate Eclipse host |
| instance). Even if tests are running on remote machines using TPTP |
| remote test execution features and deployment, the service consumers |
| machine is the machine that runs the Eclipse host instances.</p> |
| |
| <p>Thus, in the design of our automatable functional and performance |
| testing solution, functional tests are developed using AGR. TPTP |
| captures the developed test suites in instances of the TPTP Testing |
| Profile model (serialized as compressed XMI in *.testsuite files). The |
| Test Execution Service provided by TPTP Automatable Service Framework |
| (ASF) is used to execute the test suites using the command-line |
| interface that the service implementation provides.</p> |
| |
| <h2><a name="_Toc154289794">Workspace Creator</a> Application</h2> |
| |
| <p>Upon attempting to use the ASF test execution service via the |
| command-line interface/adapter one will quickly realize that one needs |
| to provide the location of an Eclipse workspace containing the test |
| suites as one of the arguments. Thus we designed and implemented a |
| workspace creator application (rich client platform application) that |
| when given an Eclipse plug-in containing the test suites |
| produces/creates an Eclipse workspace containing the test plug-in. This |
| application also provides a command-line interface/adapter in order to |
| facilitate easy invocation from within our test automation system. |
| Thus, RMPTAS gathers the plug-ins containing the test suites from source |
| control, invokes the workspace creator to create Eclipse workspaces with |
| the test plug-ins and then invokes the test execution service to execute |
| the tests. This is followed by a call to the model query service to |
| query the test execution results (test execution history serialized as |
| compressed XMI in *.execution files).</p> |
| |
| <h2><a name="_Toc154289795">Model Query Service</a></h2> |
| |
| <p>The execution of the test suites result in the creation of |
| instances of TPTP Execution History model (serialized as compressed XMI |
| in *.execution files). Execution of one test suite produces one |
| instance of the Execution History model. We implemented an Execution |
| History model query service that runs on the ASF to extract the required |
| results from the Execution History model instances. To enable the |
| RMPTAS to use the service from the command line a command-line |
| interface/adapter was implemented. The model query service queries the |
| Execution History and Testing Profile model instances to obtain all the |
| required result items (see Appendix-A for details of TPTP data models). |
| The result items that are gathered from model instances include name of |
| the test suite, names of test cases with in the test suite, test case |
| verdicts pass/fail/inconclusive and any messages (exception, stack |
| traces…etc) available from failed test cases. The time taken by |
| the system to execute each test case in a test suite can also be |
| gathered from the Execution History model instances. Presently, no |
| other performance related parameters, such as memory consumed, are |
| available in the Execution History model. However, users can monitor |
| the target host (Linux and Windows) using the TPTP monitoring tools. It |
| is understood that TPTP is working on integrating system monitoring with |
| test execution. Also, there is a feature enhancement request on AGR to |
| support gathering such values. When this feature enhancement is |
| implemented one should be able to gather other performance related data |
| from the Execution History model instances.</p> |
| |
| <p>As mentioned earlier, in the current design the results are |
| written to a log file that is read by a database population utility that |
| RMPTAS already has in place. An alternative to the design would have |
| been to use a database mutation service (based on ASF) to directly |
| populate the results in a database. However, the former was chosen in |
| order to reuse some of the functionality that was already available |
| within RMPTAS.</p> |
| |
| <h2><a name="_Toc154289796">Test Verification</a></h2> |
| |
| <p>Besides providing a mechanism to record and play back user |
| interactions with the graphical user interface, AGR provides some basic |
| test verification infrastructure. With this infrastructure, users can |
| ascertain simple things such as presence or absence of menu items using |
| AGR verification-by-selection mechanism. AGR also supports a |
| "verification hook" based test verification mechanism. To use |
| this test verification mechanism a test developer typically inserts a |
| verification point name during the test recording process (for a |
| tutorial on AGR, please see [3]). This verification point is captured |
| as a JUnit method within a verification class associated with the test |
| suite, which extends from the JUnit framework.</p> |
| |
| <p>The verification hook based verification mechanism is used in the |
| present solution design to semantically verify the contents of the |
| models produced during the execution of functional tests on IBM Rational |
| Modeling tools, such as Rational Software Architect; using a custom |
| Model Verification Library. This verification library has been designed |
| such that it can be easily invoked from within the Java-hook methods |
| with just a couple of lines of code. Test case developers may use and |
| contribute to this test verification library and/or write new |
| verification libraries for their particular test verification needs.</p> |
| |
| <p>Here is an example to further substantiate the above. In this |
| example, a user action of creating a UML Class in a modeling project is |
| recorded as an AGR test case. The system creates a class named |
| "Class1" by default. </p> |
| |
| <div class="figure"><img src="images/image002.jpg"><br> |
| Figure 2: User gesture of recording the creation of a UML Class using |
| AGR</div> |
| |
| <p>While Figure 2 shows the user gesture, Figure 3 shows the XML |
| script that is generated by AGR in the test case for this user action. |
| The reference to the verification hook method that was inserted during |
| the test case recording may be seen in the XML macro associated with |
| this test case (this is marked in red and bold in the script in Figure |
| 3). This verification hook is invoked when the test case is executed.</p> |
| |
| <div class="figure" style="margin-left:10px;margin-right:10px"> |
| <div style="text-align:left"><pre> |
| <macro version="0.1" > |
| <shell descriptive="Modeling - ModelerClassDiagram..." |
| id="org.eclipse.ui.internal.WorkbenchWindow" return-code="-1" > |
| <command descriptive="AModelingProject" type="item-select" |
| contextId="view/org.eclipse.ui.navigator.ProjectExplorer" |
| widgetId="org.eclipse.swt.widgets.Tree#{{/}}-{{1.0}}" > |
| <item path="org.eclipse.swt.widgets.TreeItem#{{/AModelingProject}}-{{1.0}}"/> |
| </command> |
| <command type="focus" contextId="view/org.eclipse.ui.navigator.ProjectExplorer" |
| widgetId="org.eclipse.swt.widgets.Composite#1" /> |
| <command descriptive="Package1" type="item-select" |
| contextId="view/org.eclipse.ui.navigator.ProjectExplorer" |
| widgetId="org.eclipse.swt.widgets.Tree#{{/}}-{{1.0}}" > |
| <item path="com.ibm.xtools.uml.navigator.ModelServerElement0|1||0||2|"/> |
| </command> |
| <command descriptive="Class" type="select" |
| contextId="popup/view/org.eclipse.ui.navigator.ProjectExplorer/org.eclipse.swt.widgets.Tree#{{/}}-{{1.0}}" |
| widgetId="org.eclipse.ui.internal.PluginActionContributionItem#{{Add &UML-&Class}}-{{0.8}}{{0|7}}-{{0.6}}{{true}}-{{0.1}}{{30}}-{{0.1}}{{Ar&tifact}}-{{0.2}}{{&Component}}-{{0.2}}" > |
| <parent widgetId="com.ibm.xtools.modeler.ui.actions.AddUMLMenu"/> |
| </command> |
| <font color="red"><strong><command type="verification" |
| contextId="view/org.eclipse.ui.navigator.ProjectExplorer" |
| location="/com.ibm.xtools.test.automation.samples.all.agrtests/src" |
| resource="com.ibm.xtools.test.automation.samples.all.agrtests.verify.ModelerClassDiagramVerifier" |
| hook="verifyClassCreation:Qorg.eclipse.ui.IViewPart;"/></strong></font> |
| <command type="wait" /> |
| </shell> |
| </macro> |
| </pre></div> |
| Figure 3: AGR test case XML macro for creating a UML Class</div> |
| <p>Figure 4 provides the details of the JUnit verification hook |
| method. With in this JUnit method, one may see the call being made to |
| the custom model verification library ModelVerifier, that is used to |
| semantically verify and assert that a UML Class named "Class1" |
| that was created during the test case execution is indeed present within |
| a UML Model named "Blank Model.emx".</p> |
| |
| |
| |
| <div class="figure"><img src="images/image003.gif"> <br> |
| Figure 4: JUnit verification hook method associated with the AGR test |
| case recording</div> |
| |
| <p>One may specify any number of verification points while recording |
| a test case, although only one is specified in the above example. If a |
| test developer chooses to do more than one concrete action within a |
| single test case, it may be beneficial to use multiple verification |
| points with each one located within the test case immediately after the |
| concrete action. This will help in quick and easy debugging of any |
| problems encountered during test case execution.</p> |
| |
| <p>The next section provides an over view of the work flow of how |
| AGR test developers in the Rational Modeling Platform Team could |
| contribute tests to be executed by RMPTAS.</p> |
| |
| <h1><a name="_Toc154289797">Test </a>Execution in RMPTAS</h1> |
| |
| <p>This section describes how developers and testers contribute |
| tests to be run by the RMPTAS. A test organization similar to that |
| specified in the AGR User Guide [3] will be followed while developing |
| the functional tests. In this test organization, a plug-in project |
| (whose name ends with a ".agrtests") holds both the AGR test |
| suites and its verification code. The test suites are contained within |
| a folder named "test-resources" and the verification code is |
| contained in a package within the "src" directory. The |
| "test-resources" folder may contain any number of test suites |
| with a corresponding verification class containing its verification hook |
| methods within the "src" folder. An example of such a |
| configuration may be found in Figure 5.</p> |
| |
| <div class="figure"><img src="images/image004.jpg"><br> |
| Figure 5: Organization of AGR Based Functional and Performance Tests</div> |
| |
| <p>The simple steps that any functional test developer would need to |
| follow for contributing tests to be executed by the RMPTAS are:</p> |
| <ul> |
| <li>Set up test development environment with AGR and Model |
| Verification library (or any other verification library that is needed) |
| and a standalone Agent Controller (although the IAC will suffice).</li> |
| |
| <li>Start the standalone Agent Controller.</li> |
| |
| <li>Create a plug-in project whose name ends with |
| ".agrtests".</li> |
| |
| <li>Create functional tests using AGR, Model verification library |
| and any other custom verification libraries.</li> |
| |
| <li>Test developed functional tests in Quick and Standard modes to |
| ensure that they execute properly.</li> |
| |
| <li>Submit code into source control.</li> |
| </ul> |
| |
| <p>RMPTAS will automatically pick up the test plug-ins from source |
| control, create entries in the database for any new tests found, execute |
| the tests and gather results.</p> |
| |
| <h1>Some Known Issues with AGR</h1> |
| |
| <p>While AGR has become much more stable than when it was originally |
| released as a Technology Preview item over a year ago, this section |
| lists some of the presently known issues in AGR. It may be worth noting |
| that while the problems listed below were found with TPTP4.2.1 and 4.3, |
| it is understood that some of the issues are slated to be resolved (or |
| already resolved) in the current release of TPTP (TPTP 4.4). </p> |
| |
| <h3>Known Problems with AGR 4.2.1 and 4.3</h3> |
| <ol> |
| <li>If the location of a widget is different between the test |
| development and test execution environments, the test cases fail to |
| execute on an "item-select" command. The default adaptive |
| widget resolver must be robust and capable of dealing with cases such |
| as this (<a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=128271">Bug |
| 128271</a>, <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=163943">Bug |
| 163943</a>). These defects have been addressed in TPTP4.3 AGR and TPTP4.4 |
| AGR. See the section on AGR in Appendix-A for details on how AGR |
| resolves items in a tree. |
| <blockquote>As an aside, it is in the authors' experience |
| that some test case developers tend to copy XML script from another |
| similar test case in order to save time instead of creating test cases |
| from scratch. This not necessarily works all the time. One of the |
| reasons is that stated as a problem in item#1 above. That is, the index |
| of the tree item that the test developer wants to work with in the new |
| test case may not necessarily be the same as the item in the test case |
| that was copied from. Note: Initial trials with TPTP4.4 AGR indicate |
| that more robust widget resolving mechanism has indeed been implemented |
| and the problem stated in Item#1 seems to have been resolved.</blockquote> |
| </li> |
| |
| <li>A similar problem is encountered due to any small change in |
| the GUI, such as when index of an item changes in a tree. In the |
| example below (see Figures 6, 7 & 8), the index of the Package |
| element "Package1" is different in the two cases. As a |
| result, for example: when one tries to select "Package1" and |
| create a UML Class element (using the test case script in Figure 3 |
| above) within it, one will quickly notice that the test (Figure 3) |
| passes in one case (Figure 6) but fails in the others (Figure 7 & |
| Figure 8). AGR needs to be fairly resilient to gracefully handle such |
| simple changes in the GUI. |
| <blockquote> |
| <div class="figure"><img src="images/image005.gif"><br> |
| Figure 6: Package Element "Package1" is the third element |
| within Blank Model</div> |
| |
| <div class="figure"><img src="images/image006.jpg"><br> |
| Figure 7: Package Element "Package1" is the first element |
| within Blank Model</div> |
| |
| <div class="figure"><img src="images/image007.jpg"><br> |
| Figure 8: Package Element "Package1" is the sixth element |
| within Blank Model</div> |
| </blockquote> |
| </li> |
| |
| <li>Difference in the state of tree items makes the test cases to |
| fail. For example: when a tree item is already open during test case |
| development, a test case developer may directly select an item |
| contained within the already open tree item. Suppose, the environment |
| where the test is executed has that tree closed; then the test case |
| fails on an "item-select" command as it is unable to select |
| the item within the closed tree. The default adaptive widget resolver |
| must be robust and capable of dealing with cases such as this (<a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=164197">Bug |
| 164197</a>). This defect has been addressed in TPTP4.4 AGR. |
| <blockquote>Workaround: A possible workaround solution is |
| to always expand the parent item first and then select the child which |
| you want to edit and modify it. Even in the case where the parent is |
| already expanded, one can close the expansion and then re-expand the |
| parent and the recording should play back successfully.</blockquote> |
| |
| <blockquote>As an aside, while it is understood that it is |
| necessary to develop the test cases in an environment that is exactly |
| the same as the one in which the test case will be executed, this is a |
| very big constraint and often very difficult to maintain.</blockquote> |
| </li> |
| <li>Inability to deal with unexpected pop-up messages (such as |
| "capability enablement prompt", for example) that does not |
| appear on the development environment while the test case is developed, |
| however it appears on the runtime environment when the test case is |
| executed. Since this is an unexpected dialog and has the focus, the |
| system does not know how to proceed further and fails during test case |
| execution (<a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=168754">Bug |
| 168754</a>). This defect has been slated to be addressed in TPTP4.4 AGR. |
| |
| <blockquote>Workaround: A possible work around for this |
| issue is to enable all capabilities up front in a separate test case |
| before the actual tests.</blockquote> |
| </li> |
| <li>The difference in the name of menu items between platforms |
| (Linux and Windows) renders AGR test cases non-portable from one |
| platform to another. For example: "File > New" on Windows |
| has a different order of shortcut keys compared to that on Linux. Due |
| to this reason, the test cases developed on one platform do not |
| necessarily execute on the other platform (<a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=170375">Bug |
| 170375</a> and <a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=159725">Bug |
| 159725</a>). The former defect has been slated to be addressed by TPTP in |
| TPTP4.4 AGR.</li> |
| |
| <li>AGR does not provide a mechanism for listening to events from |
| user interactions with native dialogs. An example of a native dialog |
| is the file dialog that appears when users click on a browse button (<a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=164192">Bug |
| 164192</a>). A target date for a fix to this problem has not yet been |
| decided.</li> |
| |
| <li>In order to minimize test recorders sensitivity to UI |
| changes, a feature called "Object Mine" was recently |
| introduced. This feature helps keep the test macro modifications to a |
| minimum when widget IDs need to be changed (<a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=144763">Bug |
| 144763</a> ). This feature enhancement was introduced in TPTP4.3 AGR</li> |
| |
| <blockquote>Note: With the object mine feature enabled, a |
| test developer needs to be cautious to not take short cuts such as |
| copy/paste test cases when recording a test case that is similar to one |
| that was recorded earlier (for example: recording the creation of a UML |
| Class versus the creation of a UML Interface in Rational Software |
| Architect). When the object mine feature is enabled, some of the common |
| information gets captured in the object mine and thus copying just the |
| information from one test case macro to another will not always work |
| correctly. To avoid any problems, it was found best not to follow any |
| such short cuts, but rather generate all test cases with appropriate |
| user gestures.</blockquote> |
| |
| <li>Very often developers develop new Eclipse plug-ins and AGR |
| test plug-ins containing test cases that test the new features provided |
| by the new plug-ins in the same workspace. In such a case, the |
| run-time environment that is launched on executing the AGR tests does |
| not contain the features provided by the new plug-in contained in the |
| development workspace. AGR needs to make it easier for users to run |
| AGR test suites against plug-ins that are included in the workspace (<a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=165526">Bug |
| 165526</a>). A target date for a fix to this problem has not yet been |
| decided.</li> |
| </ol> |
| |
| <h3>AGR Feature Enhancements:</h3> |
| |
| <ol> |
| |
| <li>AGR presently does not support GEF based recording/replay (<a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=133099">Bug |
| 133099</a> ). This feature enhancement work is in progress.</li> |
| |
| <li>AGR presently does not support drag/drop (<a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=148184"> Bug |
| 148184</a> ). This feature enhancement work is in progress.</li> |
| |
| <li>Enable automated performance testing of Eclipse applications |
| using the Automated GUI Recorder (<a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=134747">Bug |
| 134747</a> ). This feature enhancement has not yet been targeted to any |
| TPTP release.</li> |
| |
| <li>Inability to create a TPTP test suite containing different |
| types of TPTP tests such as a mixture of TPTP AGR tests, TPTP JUnit |
| tests, TPTP Manual tests… etc. (<a |
| href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=123277">Bug |
| 123277</a>). This feature enhancement has not yet been targeted to any |
| TPTP release.</li> |
| </ol> |
| |
| <h2><a name="_Toc154289798">Concluding Remarks</a></h2> |
| |
| <p>At the time of writing this paper, AGR is still a Technology |
| Preview feature and has a few issues that need to be fixed, but is seen |
| to have a very strong potential for being used as a reliable tool for |
| functional testing of Eclipse based applications. Work is underway to |
| provide GEF support in open source in order to provide a complete |
| automated testing environment. Initial trials with TPTP4.4 AGR indicate |
| that several of the problems listed in the "Some Known Issues with |
| AGR" section may have been resolved. Thus, an update to this |
| article will be issued soon after TPTP4.4 GA.</p> |
| |
| <p>Users/extenders interested in getting involved with the TPTP |
| project may find the information at the following articles useful. |
| <p> |
| <ul> |
| <li><a |
| href="http://www.eclipse.org/tptp/home/project_info/general/buildon.html">"Building |
| on TPTP"</a></li> |
| <li><a |
| href="http://www.eclipse.org/tptp/home/project_info/general/contribute_or_feedback.html">"Contributing |
| to TPTP"</a></li> |
| </ul> |
| |
| |
| <h2><a name="_Toc154289799">Acknowledgements</a></h2> |
| |
| <p>The authors wish to thank Daniel Leroux, Yves Grenier, David |
| Lander, Ali Mehregani, Scott Schneider, Paul Slauenwhite, Joseph P |
| Toomey, Christian Damus, Duc Luu and Tao Weng.</p> |
| |
| <h2><a name="_Toc154289800">Background on the authors</a></h2> |
| |
| <p>Govin Varadarajan joined IBM in 2003 with over 5 years of work |
| experience in the software industry developing network management and |
| automated resource management/provisioning software solutions. His |
| responsibilities with the IBM Rational Modeling Division include |
| development of test automation solutions, assessing and reporting on the |
| quality of modeling products. He has a Masters degree in Computer |
| Science from Queens University and two other Masters degrees in the |
| fields of Aerospace and Civil Engineering besides a Bachelor degree in |
| Civil Engineering. </p> |
| |
| <p>Kim Letkeman joined IBM in 2003 with 24 years in large financial |
| and telecommunications systems development. He is the development lead |
| for the Rational Modeling Platform team.</p> |
| |
| <h2><a name="_Toc154289801">References</a></h2> |
| |
| <p>[1] Anon, "Eclipse Test & Performance Tools Platform |
| Project (TPTP)", <a href="http://www.eclipse.org/tptp">http://www.eclipse.org/tptp</a></p> |
| |
| |
| |
| <p>[2] Anon, "Technology Preview: Automated GUI |
| Recording", <a |
| href="http://www.eclipse.org/tptp/home/downloads/downloads.php">http://www.eclipse.org/tptp/home/downloads/downloads.php</a></p> |
| |
| |
| |
| <p>[3] Anon, "TPTP AGR User Guide",</p> |
| |
| <p><a |
| href="http://www.eclipse.org/tptp/test/documents/userguides/Intro-Auto-GUI-4-2-0.html">http://www.eclipse.org/tptp/test/documents/userguides/Intro-Auto-GUI-4-2-0.html</a></p> |
| |
| <p><a |
| href="http://www.eclipse.org/tptp/test/documents/userguides/Intro-Auto-GUI-4-3-0.html">http://www.eclipse.org/tptp/test/documents/userguides/Intro-Auto-GUI-4-3-0.html</a></p> |
| |
| |
| |
| <p>[4] Chan, E and Nagarajan, G, "Profile your Java application |
| using the Eclipse Test and Performance Tools Platform (TPTP)", |
| EclipseCon 2006, <a |
| href="http://www.eclipsecon.org/2006/Sub.do?id=all&type=tutorial">http://www.eclipsecon.org/2006/Sub.do?id=all&type=tutorial</a></p> |
| |
| |
| |
| <p>[5] Yves Grenier, David Lander and Govin Varadarajan, |
| "Rational Modeling Platform Test Automation System", IBM |
| Ottawa Lab.</p> |
| |
| |
| |
| <p>[6] Kaylor, A, "The Eclipse Test and Performance Tools |
| Platform", Dr. Dobbs Portal, <a |
| href="http://www.ddj.com/184406283">http://www.ddj.com/dept/opensource/184406283</a></p> |
| |
| |
| |
| <p>[7] Schneider, S.E and Toomey, J, "Achieving Continuous |
| Integration with the Eclipse Test and Performance Tools Platform |
| (TPTP): Using TPTP Automatable Services to continually test and report |
| on the health of your system", EclipseCon 2006, <a |
| href="http://www.eclipsecon.org/2006/Sub.do?id=all&type=tutorial">http://www.eclipsecon.org/2006/Sub.do?id=all&type=tutorial</a></p> |
| |
| |
| |
| <p>[8] Slauenwhite, P, "Testing Applications Using the Eclipse |
| Test and Performance Tools Platform (TPTP)", EclipseCon 2006, <a |
| href="http://www.eclipsecon.org/2006/Sub.do?id=all&type=tutorial">http://www.eclipsecon.org/2006/Sub.do?id=all&type=tutorial</a></p> |
| |
| |
| |
| <p>[9] Slauenwhite, P, "Problem Determination of J2EE Web |
| Applications Using the Eclipse Test and Performance Tools Platform |
| (TPTP)", EclipseCon 2006, <a |
| href="http://www.eclipsecon.org/2006/Sub.do?id=all&type=tutorial">http://www.eclipsecon.org/2006/Sub.do?id=all&type=tutorial</a></p> |
| |
| |
| |
| <h1><a name="_Toc154289803">Appendix A: Technology in the |
| Functional Test Automation </a>Solution</h1> |
| |
| <p>This section provides background information on the technologies |
| used within the functional test automation solution.</p> |
| |
| <h2><a name="_Toc154289804">TPTP</a></h2> |
| |
| <p>In 2004, the Eclipse Tools project named Hyades was promoted to |
| an Eclipse Project and renamed "Eclipse Test & Performance |
| Tools Platform Project (TPTP)". TPTP is an Open-source platform for |
| Automated Software Quality tools including reference implementations for |
| testing, tracing and monitoring software systems. It addresses the |
| entire test and performance life cycle, from early testing to production |
| application monitoring, including test editing and execution, |
| monitoring, tracing, profiling, and log analysis capabilities.<b> </b>The |
| platform is designed to facilitate integration with tools used in the |
| different processes of a software lifecycle under an Eclipse environment |
| [1]. </p> |
| |
| |
| |
| <p>TPTP defines four sub projects Platform, Testing Tools, Trace |
| & Profiling and Monitoring. The sub-project that is most relevant |
| to this paper is the Testing Tools sub-project. This sub-project extends |
| the TPTP Platform and provides a common framework for testing tools |
| thereby integrating disparate test types and execution environments. It |
| includes test tool reference implementations for Manual testing, JUnit |
| and JUnit Plug-in testing, URL testing and Automated GUI |
| Recording/Playback. <a name="OLE_LINK1">[1]. </a>The TPTP platform |
| shares data through OMG-defined models. These models are described in |
| the next section.</p> |
| |
| <h2><a name="_Toc154289805">TPTP Data Models</a></h2> |
| |
| <p>TPTP implements four core EMF data models Trace, Log, |
| Statistical and Test. These models are used as an integration point |
| within TPTP. The TPTP User Interface and Execution Environment are built |
| on these data models. Model loaders consuming structured XML fragments |
| originating from a variety of sources [4, 7, 8 and 9] populate these |
| models.</p> |
| |
| |
| |
| <p>The model that is most relevant for this paper is the Test |
| model. So, only this model will be described in some detail. The Test |
| model consists of Testing Profile (see Figure 9), Behavioral and |
| Execution History data models (see Figure 10). The Testing Profile |
| data model provides a reference implementation of the UML2 Test |
| Profiles standalone MetaObject Facility model (UML2TP MOF). UML2TP |
| MOF provides a definition model for the creation, definition and |
| management of test artifacts such as Test suites, Test cases and |
| Datapools. The Behavioral model implements the UML2 interactions |
| meta-model. This model is used for defining the behavior of test suites |
| such as loops, invocations and synchronization. The Execution History |
| is a model for the creation, definition, management and persistence of |
| test executions over time. The execution of a test suite results in |
| the creation of an execution history. Execution History models include |
| information about the tests that were deployed and executed, information |
| about where it was executed (locations), verdicts, attachments, messages |
| and console output from test executions. </p> |
| |
| |
| |
| <div class="figure"><img src="images/image008.jpg"><br> |
| Figure 9: UML diagram of Testing Profile model (from [7])</div> |
| |
| <div class="figure"><img src="images/image009.jpg"><br> |
| Figure 10: UML diagram of Execution History model (from [7])</div> |
| |
| <h2><a name="_Toc154289806">TPTP Agent Controller</a></h2> |
| |
| <p>By feeding data into the models described in the above section, |
| tools open up possibilities for interaction with editors and viewers |
| that are designed around these models. The data models are supported by |
| a common framework for accessing data-collection and test-execution |
| agents. In the context of TPTP, an "agent" is defined as a |
| logical object that exposes services through the TPTP Agent Controller. |
| In TPTP, these agents run in a separate process from the Agent |
| Controller (though multiple agents can exist within a single process). </p> |
| |
| |
| |
| <p>TPTP provides a client library that manages communications with |
| an Agent Controller that can be running locally or on a remote system. |
| The Agent Controller is a process (daemon on UNIX and service on |
| Windows) that resides on each deployment host and provides the mechanism |
| by which client applications can either launch new host processes, or |
| attach to agents that coexist within existing host processes. The client |
| can reside on the same host as the Agent Controller, or it can be |
| remote. The test assets are deployed to the remote machine using the |
| Agent Controllers file transfer service. The Agent Controller provides |
| a server that has the capability to launch and manage local or remote |
| applications from a local TPTP workbench. The Agent Controller manages |
| access to and control of agents running on the target system. </p> |
| |
| |
| |
| <p>The Agent controller may be run as a stand-alone server/service |
| to deploy and execute tests either locally or remotely. TPTP workbench |
| also provides a feature called the Integrated Agent Controller (IAC) |
| that is used to launch tests locally from the TPTP workbench. The |
| biggest benefit of having Integrated Agent Controller is the |
| simplification in the usage of TPTP functionalities in the local |
| scenario by removing the dependencies on the local Agent Controller. For |
| example: there is no need to install and configure the local Agent |
| Controller when launching tests locally from the TPTP workbench. IAC |
| provides the entry-level functionalities of the standalone Agent |
| Controller. To get the full functionalities, the standalone Agent |
| Controller is still the way to go. IAC is packaged in the TPTP Platform |
| Runtime install image and therefore no separate install step is |
| required. The Integrated Agent Controller does not require any |
| configuration, but the user can configure it in the preferences page. |
| Unlike the stand-alone Agent Controller which requires the user to enter |
| information such as the path for the Java executable, the IAC will |
| determine the required information from Eclipse workbench during |
| startup.</p> |
| |
| |
| |
| <h2><a name="_Toc154289807">TPTP Automated GUI Record</a>er</h2> |
| |
| <p>TPTP Automated GUI Recorder (AGR) allows users to record and |
| playback user interactions under the Eclipse platform. The purpose of |
| the feature is to allow users to automate functional test cases for |
| applications developed under the Eclipse environment. This section |
| will only provide a quick overview of the AGR feature since a |
| well-written user guide for the same may be found on the TPTP web site |
| [3]. </p> |
| |
| |
| |
| <p>During the test case recording phase, AGR captures the user |
| gestures as XML based scripts. The so recorded script is replayed when |
| the test case is executed. AGR provides a simple "VCR-type" |
| interface to record test cases. The recorded test cases may be |
| executed either in Quick mode or Standard Mode. In Quick mode, the |
| tests are executed in the test development workbench. This is typically |
| used to quickly verify that a test case was properly recorded. In the |
| Standard mode the test cases are run on a run-time workbench using a |
| launch configuration. </p> |
| |
| |
| |
| <p>AGR supports two types of test verification mechanisms |
| verification by selection and verification via |
| "verification-hook" defined within a verification class that |
| extends from JUnit framework. Verification by selection is typically |
| used for simple verification tasks such as ensuring the presence of a |
| menu item. The idea behind this method of verification is that if the |
| system is able to select an item, then it means that the item exists. |
| Verification by verification-hook is a mechanism that uses a method |
| (hook) in a JUnit based verification class to provide the verification |
| functionality. The support for verification-hook based verification |
| basically opens the door to the use of different verification libraries |
| that users may implement. On the test case recording interface, the test |
| case developer provides the name of the verification hook method that |
| will be used to verify the test case execution.</p> |
| |
| <p>Once the test cases are defined, the user needs to define a |
| behavior that specifies how the test cases are to be executed. A |
| behavior typically includes invocations and loops. TPTP supports both |
| local and remote deployment and execution of tests. If the tests are to |
| be executed remotely the user has to define the remote context for the |
| test suite execution. Defining a context for test suite execution |
| typically requires the user to create a deployment specifying what needs |
| to be executed (the artifact containing the test suite and possible |
| datapool) and where it needs to be executed (the location). If such a |
| deployment is not created, the default is local execution. </p> |
| |
| |
| |
| <p>In order to minimize test recorders sensitivity to UI changes, |
| an optional feature that may be enabled while recording an AGR test case |
| called "Object Mine" was introduced in TPTP 4.3 AGR. This |
| feature helps keep the test macro modifications to a minimum when widget |
| IDs need to be changed. However, as mentioned in the "known |
| problems" section some problems may result when a test case |
| developer tries to copy/paste test macros with Object Mine feature |
| enabled.</p> |
| |
| |
| |
| <p>As explained in the section entitled "Widget Resolving |
| Mechanism" of the AGR User Guide [3], AGR resolves widgets based on |
| a number of properties that are defined on the widget. These properties |
| are specified in an XML file named "widgetReg.xml" that is |
| typically located at |
| "<ECLIPSE-HOME>/plugins/org.eclipse.tptp.test.auto.gui/auto-gui/". |
| This file is read by AGRs default widget resolver mechanism adaptive |
| widget resolver, to determine how a particular widget is to be |
| resolved. A weight attribute associated with each of the widget |
| property indicates how well it identifies the widget. During widget |
| resolution time AGR determines the correct widget by matching the |
| cumulative weight of a widget against the "maxThreshold" |
| attribute value provided in the widgetReg.xml file against the |
| appropriate widget class. If widgetReg.xml does not contain any |
| details for the widget in question, then default values are used for |
| widget resolution. AGR also supports a user-defined widget resolving |
| mechanism. It provides an extension point that allows clients to |
| register their own personal widget resolvers. </p> |
| |
| |
| |
| <p>It was noticed that the default widget resolution support |
| provided in TPTP4.2.0/TPTP4.2.1/TPTP4.2.2 and TPTP4.3 were insufficient |
| and resulted in problem situations as described in the "Known |
| Issues" section. However, initial trials with TPTP4.4 AGR indicate |
| that such issues have been fixed. The TPTP AGR feature is presently a |
| Technology Preview item [2]. At the time of writing this paper, some |
| suggestions have been made for making TPTP AGR feature more robust and |
| are being implemented. RCP support is impending.</p> |
| |
| |
| |
| <h2><a name="_Toc154289808">TPTP Automatable Service Framework</a></h2> |
| |
| <p>TPTP provides an Automatable Service Framework (ASF) that |
| features support for the provision and consumption of Eclipse hosted |
| TPTP services from disparate environments. Given that these services |
| are fundamentally black-box components that publish themselves via |
| extensions (with an associated specification of supported properties and |
| a behavioral contract) it is possible to create a new service provider |
| implementing the same service. This implementation approach allows for |
| a loose and dynamic binding between the service consumer and service |
| provider. Users can develop automation client adapters that adapt new |
| service consumer paradigms into the standard automation client |
| interfaces provided in TPTP. Thus, using the ASF it is possible to take |
| a program functionality that is typically limited to running inside of |
| an isolated Eclipse instance and expose it publicly to consumers outside |
| of the Eclipse environment by wrapping it with TPTP automatable service |
| abstractions [7].</p> |
| |
| |
| |
| <p>At the time of writing this paper, TPTP provides a Test Execution |
| service and various client adapters (Java, CLI and Ant) to invoke the |
| service. It also provides very rudimentary implementations of services |
| for interpreting Test Execution History models and for publishing test |
| results.</p> |
| |
| </div> |
| |
| </body> |
| |
| </html> |