| <?xml version="1.0" encoding="UTF-8"?> |
| <org.eclipse.epf.uma:ContentDescription xmi:version="2.0" |
| xmlns:xmi="http://www.omg.org/XMI" xmlns:org.eclipse.epf.uma="http://www.eclipse.org/epf/uma/1.0.4/uma.ecore" |
| xmlns:rmc="http://www.ibm.com/rmc" rmc:version="7.2.0" xmlns:epf="http://www.eclipse.org/epf" |
| epf:version="1.2.0" xmi:id="_s60KoMM3EdmSIPI87WLu3g" |
| name="test_suite,_0aDz0MlgEdmt3adZL5Dmdw" guid="_s60KoMM3EdmSIPI87WLu3g" changeDate="2006-09-29T09:48:57.425-0700" |
| version="1.0.0"> |
| <mainDescription><h3>
 |
| Introduction
 |
| </h3>
 |
| <p>
 |
| The test suite provides a means of managing the complexity of the test implementation. Many system test efforts fail
 |
| because the team gets lost in the minutia of all of the detailed tests, and subsequently loses control of the test
 |
| effort. Similar to UML packages, test suites provide a hierarchy of encapsulating containers to help manage the test
 |
| implementation. They provide a means of managing the strategic aspects of the test effort by collecting tests together
 |
| in related groups that can be planned, managed, and assessed in a meaningful way.
 |
| </p>
 |
| <p>
 |
| The test suite represents a container for organizing arbitrary collections of related test scripts. This may be
 |
| realized (implemented) as one or more automated regression test suites, but the test suite can also be a work plan for
 |
| the implementation of a group of related manual test scripts. Note also that test suites can be nested hierarchically,
 |
| therefore one test suite may be enclosed within another.
 |
| </p>
 |
| <p>
 |
| Sometimes these groups of tests will relate directly to a subsystem or other system design element, but other times
 |
| they'll relate directly to things such as quality dimensions, core "mission critical" functions, requirements
 |
| compliance, standards adherence, and many others concerns that are organized based on requirements and therefore cut
 |
| across the internal system elements.
 |
| </p>
 |
| <p>
 |
| Consider creating test suites that arrange the available test scripts, in addition to other test suites, in many
 |
| different combinations: the more variations you have, the more you'll increase coverage and the potential for finding
 |
| errors. Give thought to a variety of test suites that will cover the breadth and depth of the target test items.
 |
| Remember the corresponding implication that a single test script (or test suite) may appear in many different test
 |
| suites.
 |
| </p>
 |
| <p>
 |
| Some test automation tools provide the ability to automatically generate or assemble test suites. There are also
 |
| implementation techniques that enable automated test suites to dynamically select all or part of their component test
 |
| scripts for each test cycle run.
 |
| </p>
 |
| <p>
 |
| Test suites can help you maintain your test assets and impose a level of organization that facilitates the entire
 |
| testing effort.&nbsp; Like physical objects, tests can break. It's not that they wear down, it's that something changed
 |
| in their environment. Perhaps they've been ported to a new operating system. Or, more likely, the code they exercise
 |
| has changed in a way that correctly causes the test to fail. Suppose you're working on version 2.0 of an e-banking
 |
| application. In version 1.0, this method was used to log in:
 |
| </p>
 |
| <p class="codeSample">
 |
| public boolean login (String username);
 |
| </p>
 |
| <p>
 |
| In version 2.0, the marketing department has realized that password protection might be a good idea. So the method is
 |
| changed to this:
 |
| </p>
 |
| <p class="codeSample">
 |
| public boolean login (String username, String password);
 |
| </p>
 |
| <p>
 |
| Any test that uses the first form of the login will fail. It won't even compile. In this example you could find that,
 |
| not many useful tests can be written that don't use login. You might be faced with hundreds or thousands of failing
 |
| tests.
 |
| </p>
 |
| <p>
 |
| These tests can be fixed by using a global search-and-replace tool that finds every instance of login(something) and
 |
| replaces it with login(something, "dummy password"). Then arrange for all the testing accounts to use that password,
 |
| and you're on your way.
 |
| </p>
 |
| <p>
 |
| Then, when marketing decides that passwords should not be allowed to contain spaces, you get to do it all over again.
 |
| </p>
 |
| <p>
 |
| This kind of thing is a wasteful burden, especially when, as is often the case, the test changes aren't so easily made.
 |
| There is a better way.
 |
| </p>
 |
| <p>
 |
| Suppose that the test scripts originally did not call the product's login method. Rather, they called a library method
 |
| that does whatever it takes to get the test logged in and ready to proceed. Initially, that method might look like
 |
| this:
 |
| </p>
 |
| <p class="codeSample">
 |
| public boolean testLogin (String username) {<br />
 |
| &nbsp; return product.login(username);<br />
 |
| }
 |
| </p>
 |
| <p>
 |
| When the version 2.0 change happens, the utility library is changed to match:
 |
| </p>
 |
| <p class="codeSample">
 |
| public Boolean testLogin (String username) {<br />
 |
| &nbsp; return product.login(username, "dummy password");<br />
 |
| }
 |
| </p>
 |
| <p>
 |
| Instead of a changing a thousand tests, you change one method.
 |
| </p>
 |
| <p>
 |
| Ideally, all the needed library methods would be available at the beginning of the testing effort. In practice, they
 |
| can't all be anticipated-you might not realize you need a testLogin utility method until the first time the product
 |
| login changes. So test utility methods are often "factored out" of existing tests as needed. It is very important that
 |
| you perform this ongoing test repair, even under schedule pressure. If you do not, you will waste much time dealing
 |
| with an ugly and un-maintainable test suite. You might well find yourself throwing it away, or being unable to write
 |
| the needed numbers of new tests because all your available testing time is spent maintaining old ones.
 |
| </p>
 |
| <p>
 |
| Note: the tests of the product's login method will still call it directly. If its behavior changes, some or all of
 |
| those tests will need to be updated. (If none of the login tests fail when its behavior changes, they're probably not
 |
| very good at detecting defects.)
 |
| </p>
 |
| <h3>
 |
| Abstraction helps manage complexity
 |
| </h3>
 |
| <p>
 |
| The previous example showed how tests can abstract away from the concrete application. Most likely you can do
 |
| considerably more abstraction. You might find that a number of tests begin with a common sequence of method calls: they
 |
| log in, set up some state, and navigate to the part of the application you're testing. Only then does each test do
 |
| something different. All this setup could-and should-be contained in a single method with an evocative name such as
 |
| readyAccountForWireTransfer. By doing that, you're saving considerable time when new tests of a particular type are
 |
| written, and you're also making the intent of each test much more understandable.
 |
| </p>
 |
| <p>
 |
| Understandable tests are important. A common problem with old test suites is that no one knows what the tests are doing
 |
| or why. When they break, the tendency is to fix them in the simplest possible way. That often results in tests that are
 |
| weaker at finding defects. They no longer test what they were originally intended to test.
 |
| </p>
 |
| <h3>
 |
| Throwing away tests
 |
| </h3>
 |
| <p>
 |
| Even with utility libraries, a test might periodically be broken by behavior changes that have nothing to do with what
 |
| it checks. Fixing the test doesn't stand much of a chance of finding a defect due to the change; it's something you do
 |
| to preserve the chance of the test finding some other defect someday. But the cost of such a series of fixes might
 |
| exceed the value of the tests hypothetically finding a defect. It might be better to simply throw the test away and
 |
| devote the effort to creating new tests with greater value.
 |
| </p>
 |
| <p>
 |
| Most people resist the notion of throwing away a test, at least until they're so overwhelmed by the maintenance burden
 |
| that they throw all the tests away. It is better to make the decision carefully and continuously, test by test, asking:
 |
| </p>
 |
| <ul>
 |
| <li>
 |
| How much work will it be to fix this test well, perhaps adding to the utility library?
 |
| </li>
 |
| <li>
 |
| How else might the time be used?
 |
| </li>
 |
| <li>
 |
| How likely is it that the test will find serious defects in the future? What's been the track record of it and
 |
| related tests?
 |
| </li>
 |
| <li>
 |
| How long will it be before the test breaks again?
 |
| </li>
 |
| </ul>
 |
| <p>
 |
| The answers to these questions will be rough estimates or even guesses. But asking them will yield better results than
 |
| simply having a policy of fixing all tests.
 |
| </p>
 |
| <p>
 |
| Another reason to throw away tests is that they are now redundant. For example, early in development, there might be a
 |
| multitude of simple tests of basic parse-tree construction methods (the LessOp constructor and the like). Later, during
 |
| the writing of the parser, there will be a number of parser tests. Since the parser uses the construction methods, the
 |
| parser tests will also indirectly test them. As code changes break the construction tests, it's reasonable to discard
 |
| some of them as being redundant. Of course, any new or changed construction behavior will need new tests. They might be
 |
| implemented directly (if they're hard to test thoroughly through the parser) or indirectly (if tests through the parser
 |
| are adequate and more maintainable).
 |
| </p></mainDescription> |
| </org.eclipse.epf.uma:ContentDescription> |
