| <?xml version="1.0" encoding="UTF-8"?> |
| <org.eclipse.epf.uma:ContentDescription xmi:version="2.0" |
| xmlns:xmi="http://www.omg.org/XMI" xmlns:org.eclipse.epf.uma="http://www.eclipse.org/epf/uma/1.0.4/uma.ecore" |
| xmlns:epf="http://www.eclipse.org/epf" epf:version="1.2.0" xmi:id="-3i1jvKMUGGmAYPw4dHFbEg" |
| name="test-ideas_list,8.834380241450745E-306" guid="-3i1jvKMUGGmAYPw4dHFbEg" changeDate="2006-12-01T18:44:08.749-0500" |
| version="1.0.0"> |
| <mainDescription><h3>
 |
| <a id="Introduction" name="Introduction">Introduction</a>
 |
| </h3>
 |
| <p>
 |
| Information used in designing tests is gathered from many places: design models, classifier interfaces, statecharts,
 |
| and code itself. At some point, this source document information must be transformed into executable tests:
 |
| </p>
 |
| <ul>
 |
| <li>
 |
| specific inputs given to the software under test
 |
| </li>
 |
| <li>
 |
| in a particular hardware and software configuration
 |
| </li>
 |
| <li>
 |
| initialized to a known state
 |
| </li>
 |
| <li>
 |
| with specific results expected
 |
| </li>
 |
| </ul>
 |
| <p>
 |
| It's possible to go directly from source document information to executable tests, but it's often useful to add an
 |
| intermediate step. In this step, test ideas are written into a <i>Test-Ideas List</i>, which is used to create
 |
| executable tests.
 |
| </p>
 |
| <h3>
 |
| <a id="TestIdeas" name="TestIdeas">What are Test Ideas?</a>
 |
| </h3>
 |
| <p>
 |
| A test idea (sometimes referred to as a test requirement) is a brief statement about a test that could be performed. As
 |
| a simple example, let's consider a function that calculates a square root and come up with some test ideas:
 |
| </p>
 |
| <ul>
 |
| <li>
 |
| give a number that's barely less than zero as input
 |
| </li>
 |
| <li>
 |
| give zero as the input
 |
| </li>
 |
| <li>
 |
| test a number that's a perfect square, like 4 or 16 (is the result exactly 2 or 4?)
 |
| </li>
 |
| </ul>
 |
| <p>
 |
| Each of these ideas could readily be converted into an executable test with exact descriptions of inputs and expected
 |
| results.
 |
| </p>
 |
| <p>
 |
| There are two advantages to this less-specific intermediate form:
 |
| </p>
 |
| <ul>
 |
| <li>
 |
| test ideas are more reviewable and understandable than complete tests - it's easier to understand the reasoning
 |
| behind them
 |
| </li>
 |
| <li>
 |
| test ideas support more powerful tests, as described later under the heading <a href="#TestDesignUsingTheList">Test
 |
| Design Using the List</a>
 |
| </li>
 |
| </ul>
 |
| <p>
 |
| The square root examples all describe inputs, but test ideas can describe any of the elements of an executable test.
 |
| For example, "print to a LaserJet IIIp" describes an aspect of the test environment to be used for a test, as does
 |
| "test with database full", however, these latter test ideas are very incomplete in themselves: Print <b>what</b> to the
 |
| printer? Do <b>what</b> with that full database? They do, however, ensure that important ideas aren't forgotten; ideas
 |
| that will be described in more detail later in test design.
 |
| </p>
 |
| <p>
 |
| Test ideas are often based on fault&nbsp;models; notions of which faults are plausible in software and how those faults
 |
| can best be uncovered. For example, consider boundaries. It's safe to assume the square root function can be
 |
| implemented something like this:
 |
| </p>
 |
| <blockquote>
 |
| <pre>
 |
| double sqrt(double x) { if (x &lt; 0) // signal error ...
 |
| </pre>
 |
| </blockquote>
 |
| <p>
 |
| It's also plausible that the <font size="+0">&lt;</font> will be incorrectly typed as <font size="+0">&lt;=</font>.
 |
| People often make that kind of mistake, so it's worth checking. The fault cannot be detected with <font size="+0">X</font> having the value <font size="+0">2</font>, because both the incorrect expression (<font size="+0">x&lt;=0</font>) and the correct expression (<font size="+0">x&lt;0</font>) will take the same branch of the
 |
| <font size="+0">if</font> statement. Similarly, giving <font size="+0">X</font> the value -<font size="+0">5</font>
 |
| cannot find the fault. The only way to find it is to give <font size="+0">X</font> the value <font size="+0">0</font>,
 |
| which justifies the second test idea.
 |
| </p>
 |
| <p>
 |
| In this case, the fault model is explicit. In other cases, it's implicit. For example, whenever a program manipulates a
 |
| linked structure, it's good to test it against a circular one. It's possible that many faults could lead to a
 |
| mishandled circular structure. For the purposes of testing, they needn't be enumerated - it suffices to know that some
 |
| fault is likely enough that the test is worth running.
 |
| </p>
 |
| <p>
 |
| The following links provide information about getting test ideas from different kinds of fault models. The first two
 |
| are explicit fault models; the last uses implicit ones.
 |
| </p>
 |
| <ul>
 |
| <li>
 |
| <a class="elementLinkWithType" href="./../../../xp/guidances/guidelines/test_ideas_for_booleans_and_boundaries.html" guid="1.7150344523489172E-305">Guideline: Test Ideas for Booleans and Boundaries</a>
 |
| </li>
 |
| <li>
 |
| <a class="elementLinkWithType" href="./../../../xp/guidances/guidelines/test_ideas_for_method_calls.html" guid="8.5657170364036E-306">Guideline: Test Ideas for Method Calls</a>
 |
| </li>
 |
| <li>
 |
| <a class="elementLinkWithType" href="./../../../xp/guidances/concepts/test-ideas_catalog.html" guid="1.2384224477983028E-305">Concept: Test-Ideas Catalog</a>
 |
| </li>
 |
| </ul>
 |
| <p>
 |
| These fault models can be applied to many different artifacts. For example, the first one describes what to do with
 |
| Boolean expressions. Such expressions can be found in code, in guard conditions, in statecharts and sequence diagrams,
 |
| and in natural-language descriptions of method behaviors (such as you might find in a published API).
 |
| </p>
 |
| <p>
 |
| Occasionally it's also helpful to have guidelines for specific artifacts. See <a class="elementLinkWithType" href="./../../../xp/guidances/guidelines/test_ideas_for_statechart_and_flow_diagrams.html" guid="1.0347051690476123E-305">Guideline: Test Ideas for Statechart and Flow Diagrams</a>.
 |
| </p>
 |
| <p>
 |
| A particular Test-Ideas List might contain test ideas from many fault models, and those fault models could be derived
 |
| from more than one artifact.
 |
| </p>
 |
| <h3>
 |
| <a id="TestDesignUsingTheList" name="TestDesignUsingTheList">Test Design Using the List</a>
 |
| </h3>
 |
| <p>
 |
| Let's suppose you're designing tests for a method that searches for a string in a sequential collection. It can either
 |
| obey case or ignore case in its search, and it returns the index of the first match found or -1 if no match is found.
 |
| </p>
 |
| <blockquote>
 |
| <pre>
 |
| int Collection.find(String string, Boolean ignoreCase);
 |
| </pre>
 |
| </blockquote>
 |
| <p>
 |
| Here are some test ideas for this method:
 |
| </p>
 |
| <ol>
 |
| <li>
 |
| match found in the first position
 |
| </li>
 |
| <li>
 |
| match found in the last position
 |
| </li>
 |
| <li>
 |
| no match found
 |
| </li>
 |
| <li>
 |
| two or more matches found in the collection
 |
| </li>
 |
| <li>
 |
| case is ignored; match found, but it wouldn't match if case was obeyed
 |
| </li>
 |
| <li>
 |
| case is obeyed; an exact match is found
 |
| </li>
 |
| <li>
 |
| case is obeyed; a string that would have matched if case were ignored is skipped
 |
| </li>
 |
| </ol>
 |
| <p>
 |
| It would be simple to implement these seven tests, one for each test idea. However, different test ideas can be
 |
| combined into a single test. For example, the following test <i>satisfies</i> test ideas 2, 6, and 7:
 |
| </p>
 |
| <blockquote>
 |
| <p>
 |
| Setup: collection initialized to ["dawn", "Dawn"]<br />
 |
| Invocation: collection.find("Dawn", false)<br />
 |
| Expected result: return value is 1 (it would be 0 if "dawn" were not skipped)
 |
| </p>
 |
| </blockquote>
 |
| <p>
 |
| Making test ideas nonspecific makes them easier to combine.
 |
| </p>
 |
| <p>
 |
| It's possible to satisfy all of the test ideas in three tests. Why would three tests that satisfy seven test ideas be
 |
| better than seven separate tests?
 |
| </p>
 |
| <ul>
 |
| <li>
 |
| When you're creating a large number of simple tests, it's common to create test N+1 by copying test N and tweaking
 |
| it just enough to satisfy the new test idea. The result, especially in more complex software, is that test N+1
 |
| probably exercises the program in almost the same way as test N. It takes almost exactly the same path through the
 |
| code.<br />
 |
| <br />
 |
| A smaller number of tests, each satisfying several test ideas, doesn't allow a "copy and tweak" approach. Each
 |
| test will be somewhat different from the last, exercising the code in different ways and taking different
 |
| paths.<br />
 |
| <br />
 |
| Why would that be better? If the Test-Ideas List were complete, with a test idea for every fault in the program,
 |
| it wouldn't matter how you wrote the tests. But the list is always missing some test ideas that could find bugs. By
 |
| having each test do very different things from the last one - by adding seemingly unneeded variety - you increase
 |
| the chance that one of the tests will stumble over a bug by sheer dumb luck. In effect, smaller, more complex tests
 |
| increase the chance the test will satisfy a test idea that you didn't know you needed.<br />
 |
| </li>
 |
| <li>
 |
| Sometimes when you're creating more complex tests, new test ideas come to mind. That happens less often with simple
 |
| tests, because so much of what you're doing is exactly like the last test, which dulls your mind.
 |
| </li>
 |
| </ul>
 |
| <p>
 |
| However, there are reasons for not creating complex tests.
 |
| </p>
 |
| <ul>
 |
| <li>
 |
| If each test satisfies a single test idea and the test for idea 2 fails, you immediately know the most likely
 |
| cause: the program doesn't handle a match in the last position. If a test satisfies ideas 2, 6, and 7, then
 |
| isolating the failure is harder.<br />
 |
| </li>
 |
| <li>
 |
| Complex tests are more difficult to understand and maintain. The intent of the test is less obvious.<br />
 |
| </li>
 |
| <li>
 |
| Complex tests are more difficult to create. Constructing a test that satisfies five test ideas often takes more
 |
| time than constructing five tests that each satisfy one. Moreover, it's easier to make mistakes - to think you're
 |
| satisfying all five when you're only satisfying four.
 |
| </li>
 |
| </ul>
 |
| <p>
 |
| In practice, you must find a reasonable balance between complexity and simplicity. For example, the first tests you
 |
| subject the software to (typically the smoke tests) should be simple, easy to understand and maintain, and intended to
 |
| catch the most obvious problems. Later tests should be more complex, but not so complex they are not maintainable.
 |
| </p>
 |
| <p>
 |
| After you've finished a set of tests, it's good to check them against the characteristic test design mistakes discussed
 |
| in <a class="elementLinkWithType" href="./../../../xp/guidances/concepts/developer_testing.html#TestDesignMistakes" guid="4.085829182735815E-305">Concept: Developer Testing</a>.
 |
| </p>
 |
| <h3>
 |
| <a id="UsingTestIdeasBeforeTest" name="UsingTestIdeasBeforeTest">Using Test Ideas Before Testing</a>
 |
| </h3>
 |
| <p>
 |
| A Test-Ideas List is useful for reviews and inspections of design artifacts. For example, consider this part of a
 |
| design model showing the association between Department and Employee classes.
 |
| </p>
 |
| <p align="center">
 |
| <img height="45" alt="" src="resources/tstidslst-img1.gif" width="223" />
 |
| </p>
 |
| <p class="picturetext">
 |
| Figure 1: Association between Department and Employee Classes
 |
| </p>
 |
| <p>
 |
| The rules for creating test ideas from such a model would ask you to consider the case where a department has many
 |
| employees. By walking through a design and asking "what if, at this point, the department has many employees?", you
 |
| might discover design or analysis errors. For example, you might realize that only one employee at a time can be
 |
| transferred between departments. That might be a problem if the corporation is prone to sweeping reorganizations where
 |
| many employees need to be transferred.
 |
| </p>
 |
| <p>
 |
| Such faults, cases where a possibility was overlooked, are called <i>faults of omission</i>. Just like the faults
 |
| themselves, you have probably omitted tests that detect these faults from your testing effort. For example, see <a class="elementLinkWithUserText" href="./../../../xp/guidances/supportingmaterials/xp_and_agile_process_references.html" guid="6.191633934532389E-306">[GLA81]</a>, &nbsp;<a href="../../referenc.htm#OST84">[OST84]</a>, <a href="../../referenc.htm#BAS87">[BAS87]</a>, <a href="../../referenc.htm#MAR00">[MAR00]</a>, and other studies that
 |
| show how often faults of omission escape into deployment.
 |
| </p>
 |
| <p>
 |
| The role of testing in design activities is discussed further in <a class="elementLinkWithType" href="./../../../xp/guidances/concepts/test-first_design.html" guid="6.556259235358794E-306">Concept: Test-first Design</a>.
 |
| </p>
 |
| <h3>
 |
| <a id="TestIdeasTraceability" name="TestIdeasTraceability">Test Ideas and Traceability</a>
 |
| </h3>
 |
| <p>
 |
| Traceability is a matter of tradeoffs. Is its value worth the cost of maintaining it? This question needs to be
 |
| considered during <a href="../../activity/ac_tst_dfnasstrcnds.htm">Activity: Define Assessment and Traceability
 |
| Needs</a>.
 |
| </p>
 |
| <p>
 |
| When traceability is worthwhile, it's conventional to trace tests back to the artifacts that inspired them. For
 |
| example, you might have traceability between an API and its tests. If the API changes, you know which tests to change.
 |
| If the code (that implements the API) changes, you know which tests to run. If a test puzzles you, you can find the API
 |
| it's intended to test.
 |
| </p>
 |
| <p>
 |
| The Test-Ideas List adds another level of traceability. You can trace from a test to the test ideas it satisfies, and
 |
| then from the test ideas to the original artifact.
 |
| </p><br />
 |
| <br /></mainDescription> |
| </org.eclipse.epf.uma:ContentDescription> |
