| <?xml version="1.0" encoding="UTF-8"?> |
| <org.eclipse.epf.uma:ContentDescription xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:org.eclipse.epf.uma="http://www.eclipse.org/epf/uma/1.0.3/uma.ecore" epf:version="1.0.0" xmi:id="-3i1jvKMUGGmAYPw4dHFbEg" name="test-ideas_list,8.834380241450745E-306" guid="-3i1jvKMUGGmAYPw4dHFbEg" changeDate="2006-12-01T15:44:08.749-0800" version="1.0.0"> |
| <mainDescription><h3> |
| <a id="Introduction" name="Introduction">Introduction</a> |
| </h3> |
| <p> |
| Information used in designing tests is gathered from many places: design models, classifier interfaces, statecharts, |
| and code itself. At some point, this source document information must be transformed into executable tests: |
| </p> |
| <ul> |
| <li> |
| specific inputs given to the software under test |
| </li> |
| <li> |
| in a particular hardware and software configuration |
| </li> |
| <li> |
| initialized to a known state |
| </li> |
| <li> |
| with specific results expected |
| </li> |
| </ul> |
| <p> |
| It's possible to go directly from source document information to executable tests, but it's often useful to add an |
| intermediate step. In this step, test ideas are written into a <i>Test-Ideas List</i>, which is used to create |
| executable tests. |
| </p> |
| <h3> |
| <a id="TestIdeas" name="TestIdeas">What are Test Ideas?</a> |
| </h3> |
| <p> |
| A test idea (sometimes referred to as a test requirement) is a brief statement about a test that could be performed. As |
| a simple example, let's consider a function that calculates a square root and come up with some test ideas: |
| </p> |
| <ul> |
| <li> |
| give a number that's barely less than zero as input |
| </li> |
| <li> |
| give zero as the input |
| </li> |
| <li> |
| test a number that's a perfect square, like 4 or 16 (is the result exactly 2 or 4?) |
| </li> |
| </ul> |
| <p> |
| Each of these ideas could readily be converted into an executable test with exact descriptions of inputs and expected |
| results. |
| </p> |
| <p> |
| There are two advantages to this less-specific intermediate form: |
| </p> |
| <ul> |
| <li> |
| test ideas are more reviewable and understandable than complete tests - it's easier to understand the reasoning |
| behind them |
| </li> |
| <li> |
| test ideas support more powerful tests, as described later under the heading <a href="#TestDesignUsingTheList">Test |
| Design Using the List</a> |
| </li> |
| </ul> |
| <p> |
| The square root examples all describe inputs, but test ideas can describe any of the elements of an executable test. |
| For example, "print to a LaserJet IIIp" describes an aspect of the test environment to be used for a test, as does |
| "test with database full", however, these latter test ideas are very incomplete in themselves: Print <b>what</b> to the |
| printer? Do <b>what</b> with that full database? They do, however, ensure that important ideas aren't forgotten; ideas |
| that will be described in more detail later in test design. |
| </p> |
| <p> |
| Test ideas are often based on fault&nbsp;models; notions of which faults are plausible in software and how those faults |
| can best be uncovered. For example, consider boundaries. It's safe to assume the square root function can be |
| implemented something like this: |
| </p> |
| <blockquote> |
| <pre> |
| double sqrt(double x) { if (x &lt; 0) // signal error ... |
| </pre> |
| </blockquote> |
| <p> |
| It's also plausible that the <font size="+0">&lt;</font> will be incorrectly typed as <font size="+0">&lt;=</font>. |
| People often make that kind of mistake, so it's worth checking. The fault cannot be detected with <font |
| size="+0">X</font> having the value <font size="+0">2</font>, because both the incorrect expression (<font |
| size="+0">x&lt;=0</font>) and the correct expression (<font size="+0">x&lt;0</font>) will take the same branch of the |
| <font size="+0">if</font> statement. Similarly, giving <font size="+0">X</font> the value -<font size="+0">5</font> |
| cannot find the fault. The only way to find it is to give <font size="+0">X</font> the value <font size="+0">0</font>, |
| which justifies the second test idea. |
| </p> |
| <p> |
| In this case, the fault model is explicit. In other cases, it's implicit. For example, whenever a program manipulates a |
| linked structure, it's good to test it against a circular one. It's possible that many faults could lead to a |
| mishandled circular structure. For the purposes of testing, they needn't be enumerated - it suffices to know that some |
| fault is likely enough that the test is worth running. |
| </p> |
| <p> |
| The following links provide information about getting test ideas from different kinds of fault models. The first two |
| are explicit fault models; the last uses implicit ones. |
| </p> |
| <ul> |
| <li> |
| <a class="elementLinkWithType" |
| href="./../../../xp/guidances/guidelines/test_ideas_for_booleans_and_boundaries,1.7150344523489172E-305.html" |
| guid="1.7150344523489172E-305">Guideline: Test Ideas for Booleans and Boundaries</a> |
| </li> |
| <li> |
| <a class="elementLinkWithType" |
| href="./../../../xp/guidances/guidelines/test_ideas_for_method_calls,8.5657170364036E-306.html" |
| guid="8.5657170364036E-306">Guideline: Test Ideas for Method Calls</a> |
| </li> |
| <li> |
| <a class="elementLinkWithType" |
| href="./../../../xp/guidances/concepts/test-ideas_catalog,1.2384224477983028E-305.html" |
| guid="1.2384224477983028E-305">Concept: Test-Ideas Catalog</a> |
| </li> |
| </ul> |
| <p> |
| These fault models can be applied to many different artifacts. For example, the first one describes what to do with |
| Boolean expressions. Such expressions can be found in code, in guard conditions, in statecharts and sequence diagrams, |
| and in natural-language descriptions of method behaviors (such as you might find in a published API). |
| </p> |
| <p> |
| Occasionally it's also helpful to have guidelines for specific artifacts. See <a class="elementLinkWithType" |
| href="./../../../xp/guidances/guidelines/test_ideas_for_statechart_and_flow_diagrams,1.0347051690476123E-305.html" |
| guid="1.0347051690476123E-305">Guideline: Test Ideas for Statechart and Flow Diagrams</a>. |
| </p> |
| <p> |
| A particular Test-Ideas List might contain test ideas from many fault models, and those fault models could be derived |
| from more than one artifact. |
| </p> |
| <h3> |
| <a id="TestDesignUsingTheList" name="TestDesignUsingTheList">Test Design Using the List</a> |
| </h3> |
| <p> |
| Let's suppose you're designing tests for a method that searches for a string in a sequential collection. It can either |
| obey case or ignore case in its search, and it returns the index of the first match found or -1 if no match is found. |
| </p> |
| <blockquote> |
| <pre> |
| int Collection.find(String string, Boolean ignoreCase); |
| </pre> |
| </blockquote> |
| <p> |
| Here are some test ideas for this method: |
| </p> |
| <ol> |
| <li> |
| match found in the first position |
| </li> |
| <li> |
| match found in the last position |
| </li> |
| <li> |
| no match found |
| </li> |
| <li> |
| two or more matches found in the collection |
| </li> |
| <li> |
| case is ignored; match found, but it wouldn't match if case was obeyed |
| </li> |
| <li> |
| case is obeyed; an exact match is found |
| </li> |
| <li> |
| case is obeyed; a string that would have matched if case were ignored is skipped |
| </li> |
| </ol> |
| <p> |
| It would be simple to implement these seven tests, one for each test idea. However, different test ideas can be |
| combined into a single test. For example, the following test <i>satisfies</i> test ideas 2, 6, and 7: |
| </p> |
| <blockquote> |
| <p> |
| Setup: collection initialized to ["dawn", "Dawn"]<br /> |
| Invocation: collection.find("Dawn", false)<br /> |
| Expected result: return value is 1 (it would be 0 if "dawn" were not skipped) |
| </p> |
| </blockquote> |
| <p> |
| Making test ideas nonspecific makes them easier to combine. |
| </p> |
| <p> |
| It's possible to satisfy all of the test ideas in three tests. Why would three tests that satisfy seven test ideas be |
| better than seven separate tests? |
| </p> |
| <ul> |
| <li> |
| When you're creating a large number of simple tests, it's common to create test N+1 by copying test N and tweaking |
| it just enough to satisfy the new test idea. The result, especially in more complex software, is that test N+1 |
| probably exercises the program in almost the same way as test N. It takes almost exactly the same path through the |
| code.<br /> |
| <br /> |
| A smaller number of tests, each satisfying several test ideas, doesn't allow a "copy and tweak" approach. Each |
| test will be somewhat different from the last, exercising the code in different ways and taking different |
| paths.<br /> |
| <br /> |
| Why would that be better? If the Test-Ideas List were complete, with a test idea for every fault in the program, |
| it wouldn't matter how you wrote the tests. But the list is always missing some test ideas that could find bugs. By |
| having each test do very different things from the last one - by adding seemingly unneeded variety - you increase |
| the chance that one of the tests will stumble over a bug by sheer dumb luck. In effect, smaller, more complex tests |
| increase the chance the test will satisfy a test idea that you didn't know you needed.<br /> |
| </li> |
| <li> |
| Sometimes when you're creating more complex tests, new test ideas come to mind. That happens less often with simple |
| tests, because so much of what you're doing is exactly like the last test, which dulls your mind. |
| </li> |
| </ul> |
| <p> |
| However, there are reasons for not creating complex tests. |
| </p> |
| <ul> |
| <li> |
| If each test satisfies a single test idea and the test for idea 2 fails, you immediately know the most likely |
| cause: the program doesn't handle a match in the last position. If a test satisfies ideas 2, 6, and 7, then |
| isolating the failure is harder.<br /> |
| </li> |
| <li> |
| Complex tests are more difficult to understand and maintain. The intent of the test is less obvious.<br /> |
| </li> |
| <li> |
| Complex tests are more difficult to create. Constructing a test that satisfies five test ideas often takes more |
| time than constructing five tests that each satisfy one. Moreover, it's easier to make mistakes - to think you're |
| satisfying all five when you're only satisfying four. |
| </li> |
| </ul> |
| <p> |
| In practice, you must find a reasonable balance between complexity and simplicity. For example, the first tests you |
| subject the software to (typically the smoke tests) should be simple, easy to understand and maintain, and intended to |
| catch the most obvious problems. Later tests should be more complex, but not so complex they are not maintainable. |
| </p> |
| <p> |
| After you've finished a set of tests, it's good to check them against the characteristic test design mistakes discussed |
| in <a class="elementLinkWithType" |
| href="./../../../xp/guidances/concepts/developer_testing,4.085829182735815E-305.html#TestDesignMistakes" |
| guid="4.085829182735815E-305">Concept: Developer Testing</a>. |
| </p> |
| <h3> |
| <a id="UsingTestIdeasBeforeTest" name="UsingTestIdeasBeforeTest">Using Test Ideas Before Testing</a> |
| </h3> |
| <p> |
| A Test-Ideas List is useful for reviews and inspections of design artifacts. For example, consider this part of a |
| design model showing the association between Department and Employee classes. |
| </p> |
| <p align="center"> |
| <img height="45" alt="" src="resources/tstidslst-img1.gif" width="223" /> |
| </p> |
| <p class="picturetext"> |
| Figure 1: Association between Department and Employee Classes |
| </p> |
| <p> |
| The rules for creating test ideas from such a model would ask you to consider the case where a department has many |
| employees. By walking through a design and asking "what if, at this point, the department has many employees?", you |
| might discover design or analysis errors. For example, you might realize that only one employee at a time can be |
| transferred between departments. That might be a problem if the corporation is prone to sweeping reorganizations where |
| many employees need to be transferred. |
| </p> |
| <p> |
| Such faults, cases where a possibility was overlooked, are called <i>faults of omission</i>. Just like the faults |
| themselves, you have probably omitted tests that detect these faults from your testing effort. For example, see <a |
| class="elementLinkWithUserText" |
| href="./../../../xp/guidances/supportingmaterials/xp_and_agile_process_references,6.191633934532389E-306.html" |
| guid="6.191633934532389E-306">[GLA81]</a>, &nbsp;<a href="../../referenc.htm#OST84">[OST84]</a>, <a |
| href="../../referenc.htm#BAS87">[BAS87]</a>, <a href="../../referenc.htm#MAR00">[MAR00]</a>, and other studies that |
| show how often faults of omission escape into deployment. |
| </p> |
| <p> |
| The role of testing in design activities is discussed further in <a class="elementLinkWithType" |
| href="./../../../xp/guidances/concepts/test-first_design,6.556259235358794E-306.html" |
| guid="6.556259235358794E-306">Concept: Test-first Design</a>. |
| </p> |
| <h3> |
| <a id="TestIdeasTraceability" name="TestIdeasTraceability">Test Ideas and Traceability</a> |
| </h3> |
| <p> |
| Traceability is a matter of tradeoffs. Is its value worth the cost of maintaining it? This question needs to be |
| considered during <a href="../../activity/ac_tst_dfnasstrcnds.htm">Activity: Define Assessment and Traceability |
| Needs</a>. |
| </p> |
| <p> |
| When traceability is worthwhile, it's conventional to trace tests back to the artifacts that inspired them. For |
| example, you might have traceability between an API and its tests. If the API changes, you know which tests to change. |
| If the code (that implements the API) changes, you know which tests to run. If a test puzzles you, you can find the API |
| it's intended to test. |
| </p> |
| <p> |
| The Test-Ideas List adds another level of traceability. You can trace from a test to the test ideas it satisfies, and |
| then from the test ideas to the original artifact. |
| </p> |
| <br /> |
| <br /></mainDescription> |
| </org.eclipse.epf.uma:ContentDescription> |