| <?xml version="1.0" encoding="UTF-8"?> |
| <org.eclipse.epf.uma:ContentDescription xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:org.eclipse.epf.uma="http://www.eclipse.org/epf/uma/1.0.3/uma.ecore" epf:version="1.0.0" xmi:id="_s60KoMM3EdmSIPI87WLu3g" name="test_suite,_0aDz0MlgEdmt3adZL5Dmdw" guid="_s60KoMM3EdmSIPI87WLu3g" changeDate="2006-09-29T12:48:57.425-0400" version="1.0.0"> |
| <mainDescription><h3> |
| Introduction |
| </h3> |
| <p> |
| The test suite provides a means of managing the complexity of the test implementation. Many system test efforts fail |
| because the team gets lost in the minutia of all of the detailed tests, and subsequently loses control of the test |
| effort. Similar to UML packages, test suites provide a hierarchy of encapsulating containers to help manage the test |
| implementation. They provide a means of managing the strategic aspects of the test effort by collecting tests together |
| in related groups that can be planned, managed, and assessed in a meaningful way. |
| </p> |
| <p> |
| The test suite represents a container for organizing arbitrary collections of related test scripts. This may be |
| realized (implemented) as one or more automated regression test suites, but the test suite can also be a work plan for |
| the implementation of a group of related manual test scripts. Note also that test suites can be nested hierarchically, |
| therefore one test suite may be enclosed within another. |
| </p> |
| <p> |
| Sometimes these groups of tests will relate directly to a subsystem or other system design element, but other times |
| they'll relate directly to things such as quality dimensions, core "mission critical" functions, requirements |
| compliance, standards adherence, and many others concerns that are organized based on requirements and therefore cut |
| across the internal system elements. |
| </p> |
| <p> |
| Consider creating test suites that arrange the available test scripts, in addition to other test suites, in many |
| different combinations: the more variations you have, the more you'll increase coverage and the potential for finding |
| errors. Give thought to a variety of test suites that will cover the breadth and depth of the target test items. |
| Remember the corresponding implication that a single test script (or test suite) may appear in many different test |
| suites. |
| </p> |
| <p> |
| Some test automation tools provide the ability to automatically generate or assemble test suites. There are also |
| implementation techniques that enable automated test suites to dynamically select all or part of their component test |
| scripts for each test cycle run. |
| </p> |
| <p> |
| Test suites can help you maintain your test assets and impose a level of organization that facilitates the entire |
| testing effort.&nbsp; Like physical objects, tests can break. It's not that they wear down, it's that something changed |
| in their environment. Perhaps they've been ported to a new operating system. Or, more likely, the code they exercise |
| has changed in a way that correctly causes the test to fail. Suppose you're working on version 2.0 of an e-banking |
| application. In version 1.0, this method was used to log in: |
| </p> |
| <p class="codeSample"> |
| public boolean login (String username); |
| </p> |
| <p> |
| In version 2.0, the marketing department has realized that password protection might be a good idea. So the method is |
| changed to this: |
| </p> |
| <p class="codeSample"> |
| public boolean login (String username, String password); |
| </p> |
| <p> |
| Any test that uses the first form of the login will fail. It won't even compile. In this example you could find that, |
| not many useful tests can be written that don't use login. You might be faced with hundreds or thousands of failing |
| tests. |
| </p> |
| <p> |
| These tests can be fixed by using a global search-and-replace tool that finds every instance of login(something) and |
| replaces it with login(something, "dummy password"). Then arrange for all the testing accounts to use that password, |
| and you're on your way. |
| </p> |
| <p> |
| Then, when marketing decides that passwords should not be allowed to contain spaces, you get to do it all over again. |
| </p> |
| <p> |
| This kind of thing is a wasteful burden, especially when, as is often the case, the test changes aren't so easily made. |
| There is a better way. |
| </p> |
| <p> |
| Suppose that the test scripts originally did not call the product's login method. Rather, they called a library method |
| that does whatever it takes to get the test logged in and ready to proceed. Initially, that method might look like |
| this: |
| </p> |
| <p class="codeSample"> |
| public boolean testLogin (String username) {<br /> |
| &nbsp; return product.login(username);<br /> |
| } |
| </p> |
| <p> |
| When the version 2.0 change happens, the utility library is changed to match: |
| </p> |
| <p class="codeSample"> |
| public Boolean testLogin (String username) {<br /> |
| &nbsp; return product.login(username, "dummy password");<br /> |
| } |
| </p> |
| <p> |
| Instead of a changing a thousand tests, you change one method. |
| </p> |
| <p> |
| Ideally, all the needed library methods would be available at the beginning of the testing effort. In practice, they |
| can't all be anticipated-you might not realize you need a testLogin utility method until the first time the product |
| login changes. So test utility methods are often "factored out" of existing tests as needed. It is very important that |
| you perform this ongoing test repair, even under schedule pressure. If you do not, you will waste much time dealing |
| with an ugly and un-maintainable test suite. You might well find yourself throwing it away, or being unable to write |
| the needed numbers of new tests because all your available testing time is spent maintaining old ones. |
| </p> |
| <p> |
| Note: the tests of the product's login method will still call it directly. If its behavior changes, some or all of |
| those tests will need to be updated. (If none of the login tests fail when its behavior changes, they're probably not |
| very good at detecting defects.) |
| </p> |
| <h3> |
| Abstraction helps manage complexity |
| </h3> |
| <p> |
| The previous example showed how tests can abstract away from the concrete application. Most likely you can do |
| considerably more abstraction. You might find that a number of tests begin with a common sequence of method calls: they |
| log in, set up some state, and navigate to the part of the application you're testing. Only then does each test do |
| something different. All this setup could-and should-be contained in a single method with an evocative name such as |
| readyAccountForWireTransfer. By doing that, you're saving considerable time when new tests of a particular type are |
| written, and you're also making the intent of each test much more understandable. |
| </p> |
| <p> |
| Understandable tests are important. A common problem with old test suites is that no one knows what the tests are doing |
| or why. When they break, the tendency is to fix them in the simplest possible way. That often results in tests that are |
| weaker at finding defects. They no longer test what they were originally intended to test. |
| </p> |
| <h3> |
| Throwing away tests |
| </h3> |
| <p> |
| Even with utility libraries, a test might periodically be broken by behavior changes that have nothing to do with what |
| it checks. Fixing the test doesn't stand much of a chance of finding a defect due to the change; it's something you do |
| to preserve the chance of the test finding some other defect someday. But the cost of such a series of fixes might |
| exceed the value of the tests hypothetically finding a defect. It might be better to simply throw the test away and |
| devote the effort to creating new tests with greater value. |
| </p> |
| <p> |
| Most people resist the notion of throwing away a test, at least until they're so overwhelmed by the maintenance burden |
| that they throw all the tests away. It is better to make the decision carefully and continuously, test by test, asking: |
| </p> |
| <ul> |
| <li> |
| How much work will it be to fix this test well, perhaps adding to the utility library? |
| </li> |
| <li> |
| How else might the time be used? |
| </li> |
| <li> |
| How likely is it that the test will find serious defects in the future? What's been the track record of it and |
| related tests? |
| </li> |
| <li> |
| How long will it be before the test breaks again? |
| </li> |
| </ul> |
| <p> |
| The answers to these questions will be rough estimates or even guesses. But asking them will yield better results than |
| simply having a policy of fixing all tests. |
| </p> |
| <p> |
| Another reason to throw away tests is that they are now redundant. For example, early in development, there might be a |
| multitude of simple tests of basic parse-tree construction methods (the LessOp constructor and the like). Later, during |
| the writing of the parser, there will be a number of parser tests. Since the parser uses the construction methods, the |
| parser tests will also indirectly test them. As code changes break the construction tests, it's reasonable to discard |
| some of them as being redundant. Of course, any new or changed construction behavior will need new tests. They might be |
| implemented directly (if they're hard to test thoroughly through the parser) or indirectly (if tests through the parser |
| are adequate and more maintainable). |
| </p></mainDescription> |
| </org.eclipse.epf.uma:ContentDescription> |