| <?xml version="1.0" encoding="UTF-8"?> |
| <org.eclipse.epf.uma:ContentDescription xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:org.eclipse.epf.uma="http://www.eclipse.org/epf/uma/1.0.3/uma.ecore" epf:version="1.0.0" xmi:id="-3AbfvnHrCOIQS63sEjrOew" name="test-first_design,6.556259235358794E-306" guid="-3AbfvnHrCOIQS63sEjrOew" changeDate="2006-11-13T15:58:35.617-0800" version="1.0.0"> |
| <mainDescription><a id="XE_test__developer_testing__test-first_design" name="XE_test__developer_testing__test-first_design"></a><a |
| id="XE_design__test-first_design" name="XE_design__test-first_design"></a> |
| <h3> |
| <a id="Introduction" name="Introduction">Introduction</a> |
| </h3> |
| <p> |
| Test designs are created using information from a variety of artifacts, including design artifacts such as use case |
| realizations, design models, or classifier interfaces. Tests are executed after components are created. It's typical to |
| create the test designs just before the tests are to be executed - well after the software design artifacts are |
| created. Figure 1, following, shows an example. Here, test design begins sometime toward the end of implementation. It |
| draws on the results of component design. The arrow from Implementation to Test Execution indicates that the tests |
| can't be executed until the implementation is complete. |
| </p> |
| <p align="center"> |
| <img height="159" alt="" src="resources/tstfrsdsg-img1.gif" width="614" /> |
| </p> |
| <p class="picturetext"> |
| Fig1: Traditionally, Test Design is performed later in the life-cycle |
| </p> |
| <p> |
| However, it doesn't have to be this way. Although test execution has to wait until the component has been implemented, |
| test design can be done earlier. It could be done just after the design artifact is completed. It could even be done in |
| parallel with component design, as shown here: |
| </p> |
| <p align="center"> |
| <img height="158" alt="" src="resources/tstfrsdsg-img2.gif" width="610" /> |
| </p> |
| <p class="picturetext"> |
| Fig2: Test-first Design brings test design chronologically in-line with software design |
| </p> |
| <p> |
| Moving the test effort "upstream" in this way is commonly called "test-first design". What are its advantages? |
| </p> |
| <ol> |
| <li> |
| No matter how carefully you design software, you'll make mistakes. You might be missing a relevant fact. Or you |
| might have particular habits of thought that make it hard for you to see certain alternatives. Or you might just be |
| tired and overlook something. Having other people review your design artifacts helps. They might have the facts you |
| miss, or they might see what you overlooked. It's best if these people have a different perspective than you do; by |
| looking at the design differently, they'll see things you missed.<br /> |
| <br /> |
| Experience has shown that the testing perspective is an effective one. It's relentlessly concrete. During software |
| design, it's easy to think of a particular field as "displaying the title of the current customer" and move on |
| without really thinking about it. During test design, you must decide <i>specifically</i> what that field will show |
| when a customer who retired from the Navy and then obtained a law degree insists on referring to himself as |
| "Lieutenant Morton H. Throckbottle (Ret.), Esq." Is his title "Lieutenant" or "Esquire"?<br /> |
| <br /> |
| If test design is deferred until just before test execution, as in Figure 1, you'll probably waste money. A |
| mistake in your software design will remain uncaught until test design, when some tester says, "You know, I knew |
| this guy from the Navy...", creates the "Morton" test, and discovers the problem. Now a partially or fully complete |
| implementation has to be rewritten and a design artifact has to be updated. It would be cheaper to catch the |
| problem before implementation begins.<br /> |
| </li> |
| <li> |
| Some mistakes might be caught before test design. Instead, they'll be caught by the Implementer. That's still bad. |
| Implementation must grind to a halt while the focus switches from how to implement the design to what that design |
| should be. That's disruptive even when the Implementer and Designer roles are filled by the same person; it's much |
| more disruptive when they're different people. Preventing this disruption is another way in which test-first design |
| helps improve efficiency.<br /> |
| </li> |
| <li> |
| Test designs help Implementers in another way-by clarifying design. If there's a question in the Implementer's mind |
| about what the design meant, the test design might serve as a specific example of the desired behavior. That will |
| lead to fewer bugs due to Implementer misunderstanding.<br /> |
| </li> |
| <li> |
| There are fewer bugs even if the question <i>wasn't</i> in the Implementer's mind - but should have been. For |
| example, there might have been an ambiguity that the Designer unconsciously interpreted one way and the Implementer |
| another. If the Implementer is working from both the design and also from specific instructions about what the |
| component is supposed to do - from test cases - the component is more likely to actually do what is required. |
| </li> |
| </ol> |
| <h3> |
| <a id="Examples" name="Examples">Examples</a> |
| </h3> |
| <p> |
| Here are some examples to give you the flavor of test-first design. |
| </p> |
| <p> |
| Suppose you're creating a system to replace the old "ask the secretary" method of assigning meeting rooms. One of the |
| methods of the <font size="+0">MeetingDatabase</font> class is called <font size="+0">getMeeting</font>, which has this |
| signature: |
| </p> |
| <blockquote> |
| <pre> |
| Meeting getMeeting(Person, Time); |
| </pre> |
| </blockquote> |
| <p> |
| Given a person and a time, <font size="+0">getMeeting</font> returns the meeting that person is scheduled to be in at |
| that time. If the person isn't scheduled for anything, it returns the special <font size="+0">Meeting</font> object |
| <font size="+0">unscheduled</font>. There are some straightforward test cases: |
| </p> |
| <ul> |
| <li> |
| The person isn't in any meeting at the given time. Is the <font size="+0">unscheduled</font> meeting returned? |
| </li> |
| <li> |
| The person is in a meeting at that time. Does the method return the correct meeting? |
| </li> |
| </ul> |
| <p> |
| These test cases are unexciting, but they need to be tried eventually. They might as well be created now, by writing |
| the actual test code that will someday be run. Java code for the first test might look like this: |
| </p> |
| <pre> |
| // if not in a meeting at given time, |
| // expect to be unscheduled. |
| public void testWhenAvailable() { |
| Person fred = new Person("fred"); |
| Time now = Time.now(); |
| MeetingDatabase db = new MeetingDatabase(); |
| expect(db.getMeeting(fred, now) == Meeting.unscheduled); |
| } |
| </pre> |
| <p> |
| But there are more interesting test ideas. For example, this method searches for a match. Whenever a method searches, |
| it's a good idea to ask what should happen if the search finds more than one match. In this case, that means asking |
| "Can a person be in two meetings at once?" Seems impossible, but asking the secretary about that case might reveal |
| something surprising. It turns out that some executives are quite often scheduled into two meetings at once. Their role |
| is to pop into a meeting, "rally the troops" for some short amount of time, and then move on. A system that didn't |
| accommodate that behavior would go at least partially unused. |
| </p> |
| <p> |
| This is an example of test-first design done at the implementation level catching an analysis problem. There are a few |
| things to note about that: |
| </p> |
| <ol> |
| <li> |
| You would hope that good use-case specification and analysis would have already discovered this requirement. In |
| that case, the problem would have been avoided "upstream" and <font size="+0">getMeeting</font> would have been |
| designed differently. (It couldn't return a meeting; it would have to return a set of meetings.) But analysis |
| always misses some problems, and it's better for them to be discovered during implementation than after |
| deployment.<br /> |
| </li> |
| <li> |
| In many cases, Designers and Implementers won't have the domain knowledge to catch such problems - they won't have |
| the opportunity or time to quiz the secretary. In that case, the person designing tests for <font |
| size="+0">getMeeting</font> would ask, "is there a case in which two meetings should be returned?", think for a |
| while, and conclude that there wasn't. So test-first design doesn't catch every problem, but the mere fact of |
| asking the right kinds of questions increases the chance a problem will be found.<br /> |
| </li> |
| <li> |
| Some of the same testing techniques that apply during implementation also apply to analysis. Test-first design can |
| be done by analysts as well, but that's not the topic of this page. |
| </li> |
| </ol> |
| <p> |
| The second of the three examples is a statechart model for a heating system. |
| </p> |
| <p align="center"> |
| <img height="253" alt="" src="resources/tstfrsdsg-img3.gif" width="567" /> |
| </p> |
| <p class="picturetext"> |
| Fig3: HVAC Statechart |
| </p> |
| <p> |
| A set of tests would traverse all the arcs in the statechart. One test might begin with an idle system, inject a Too |
| Hot event, fail the system during the Cooling/Running state, clear the failure, inject another Too Hot event, then run |
| the system back to the Idle state. Since that does not exercise all the arcs, more tests are needed. These kinds of |
| tests look for various kinds of implementation problems. For example, by traversing every arc, they check whether the |
| implementation has left one out. By using sequences of events that have failure paths followed by paths that should |
| successfully complete, they check whether error-handling code fails to clean up partial results that might affect later |
| computation. (For more about testing statecharts, see <a class="elementLinkWithUserText" |
| href="./../../../xp/guidances/guidelines/test_ideas_for_statechart_and_flow_diagrams,1.0347051690476123E-305.html" |
| guid="1.0347051690476123E-305">Guideline: Test Ideas for Statechart and Activity Diagrams</a>.) |
| </p> |
| <p> |
| The final example uses part of a design model. There's an association between a creditor and an invoice, where any |
| given creditor can have more than one invoice outstanding. |
| </p> |
| <p align="center"> |
| <img height="45" alt="" src="resources/tstfrsdsg-img4.gif" width="186" /> |
| </p> |
| <p class="picturetext"> |
| Fig4: Association between Creditor and Invoice Classes |
| </p> |
| <p> |
| Tests based on this model would exercise the system when a creditor has no invoices, one invoice, and a large number of |
| invoices. A tester would also ask whether there are situations in which an invoice might need to be associated with |
| more than one creditor, or where an invoice has no creditor. (Perhaps the people who currently run the paper-based |
| system the computer system is to replace use creditor-less invoices as a way to keep track of pending work). If so, |
| that would be another problem that should have been caught in Analysis. |
| </p> |
| <h3> |
| <a id="WhoDoesTest-FirstDesign" name="WhoDoesTest-FirstDesign">Who does test-first design?</a> |
| </h3> |
| <p> |
| Test-first design can be done by either the author of the design or by someone else. It's common for the author to do |
| it. The advantage is that it reduces communication overhead. The Designer and Test Designer don't have to explain |
| things to each other. Further, a separate Test Designer would have to spend time learning the design well, whereas the |
| original Designer already knows it. Finally, many of these questions - like "what happens if the compressor fails in |
| state X?" - are natural questions to ask during both software artifact design and test design, so you might as well |
| have the same person ask them exactly once and write the answers down in the form of tests. |
| </p> |
| <p> |
| There are disadvantages, though. The first is that the Designer is, to some extent, blind to his or her own mistakes. |
| The test design process will reveal some of that blindness, but probably not as much as a different person would find. |
| How much of a problem this is seems to vary widely from person to person and is often related to the amount of |
| experience the Designer has. |
| </p> |
| <p> |
| Another disadvantage of having the same person do both software design and test design is that there's no parallelism. |
| Whereas allocating the roles to separate people will take more total effort, it will probably result in less elapsed |
| calendar time. If people are itching to move out of design and into implementation, taking time for test design can be |
| frustrating. More importantly, there's a tendency to skimp on the work in order to move on. |
| </p> |
| <h3> |
| <a id="CanAllTestDesignBeDoneAtComponentDesignTime" name="CanAllTestDesignBeDoneAtComponentDesignTime">Can all test |
| design be done at component design time?</a> |
| </h3> |
| <p> |
| No. The reason is that not all decisions are made at design time. Decisions made during implementation won't be |
| well-tested by tests created from the design. The classic example of this is a routine to sort arrays. There are many |
| different sorting algorithms with different tradeoffs. Quicksort is usually faster than an insertion sort on large |
| arrays, but often slower on small arrays. So a sorting algorithm might be implemented to use Quicksort for arrays with |
| more than 15 elements, but insertion sort otherwise. That division of labor might be invisible from design artifacts. |
| You <i>could</i> represent it in a design artifact, but the Designer might have decided that the benefit of making such |
| explicit decisions wasn't worthwhile. Since the size of the array plays no role in the design, the test design might |
| inadvertently use only small arrays, meaning that none of the Quicksort code would be tested at all. |
| </p> |
| <p> |
| As another example, consider this fraction of a sequence diagram. It shows a <font size="+0">SecurityManager</font> |
| calling the <font size="+0">log()</font> method of <font size="+0">StableStore</font>. In this case, though, the <font |
| size="+0">log()</font> returns a failure, which causes <font size="+0">SecurityManager</font> to call <font |
| size="+0">Connection.close()</font>. |
| </p> |
| <p align="center"> |
| <img height="161" alt="" src="resources/tstfrsdsg-img5.gif" width="303" /> |
| </p> |
| <p class="picturetext"> |
| Fig5: SecurityManager sequence diagram instance |
| </p> |
| <p> |
| This is a good reminder to the Implementer. Whenever <font size="+0">log()</font> fails, the connection must be closed. |
| The question for testing to answer is whether the Implementer really did it-and did it correctly-in <i>all</i> cases or |
| just in <i>some</i>. To answer the question, the Test Designer must find all the calls to <font |
| size="+0">StableStore.log()</font> and make sure each of those call points is given a failure to handle. |
| </p> |
| <p> |
| It might seem odd to run such a test, given that you've just looked at all the code that calls <font |
| size="+0">StableStore.log()</font>. Can't you just check to see if it handles failure correctly? |
| </p> |
| <p> |
| Perhaps inspection might be enough. But error-handling code is notoriously error-prone because it often implicitly |
| depends on assumptions that the existence of the error has violated. The classic example of this is code that handles |
| allocation failures. Here's an example: |
| </p> |
| <blockquote> |
| <pre> |
| while (true) { // top level event loop |
| try { |
| XEvent xe = getEvent(); |
| ... // main body of program |
| } catch (OutOfMemoryError e) { |
| emergencyRestart(); |
| } |
| } |
| </pre> |
| </blockquote> |
| <p> |
| This code attempts to recover from out of memory errors by cleaning up (thus making memory available) and then |
| continuing to process events. Let's suppose that's an acceptable design. <font size="+0">emergencyRestart</font> takes |
| great care not to allocate memory. The problem is that <font size="+0">emergencyRestart</font> calls some utility |
| routine, which calls some other utility routine, which calls some other utility routine-which allocates a new object. |
| Except that there's no memory, so the whole program fails. These kinds of problems are hard to find through inspection. |
| </p> |
| <h3> |
| <a id="Test-FirstDesignAndRUPPhases" name="Test-FirstDesignAndRUPPhases">Test-first design and the phases |
| of&nbsp;Unified</a> Process |
| </h3> |
| <p> |
| Up to this point, we've implicitly assumed that you'd do as much test design as possible as early as possible. That is, |
| you'd derive all the tests you could from the design artifact, later adding only tests based on implementation |
| internals. That may not be appropriate in the Elaboration phase, because such complete testing may not be aligned with |
| an iteration's objectives. |
| </p> |
| <p> |
| Suppose an architectural prototype is being built to demonstrate product feasibility to investors. It might be based on |
| a few key use-case instances. Code should be tested to see that it supports them. But is there any harm if further |
| tests are created? For example, it might be obvious that the prototype ignores important error cases. Why not document |
| the need for that error handling by writing test cases that will exercise it? |
| </p> |
| <p> |
| But what if the prototype does its job and reveals that the architectural approach won't work? Then the architecture |
| will be thrown away - along with all those tests for error-handling. In that case, the effort of designing the tests |
| will have yielded no value. It would have been better to have waited, and only designed those tests needed to check |
| whether this proof-of-concept prototype really proves the concept. |
| </p> |
| <p> |
| This may seem a minor point, but there are strong psychological effects in play. The Elaboration phase is about |
| addressing major risks. The whole project team should be focused on those risks. Having people concentrating on minor |
| issues drains focus and energy from the team. |
| </p> |
| <p> |
| So where might test-first design be used successfully in the Elaboration phase? It can play an important role in |
| adequately exploring architectural risks. Considering how, precisely, the team will know if a risk has been realized or |
| avoided will add clarity to the design process and may well result in a better architecture being built the first time. |
| </p> |
| <p> |
| During the Construction phase, design artifacts are put into their final form. All the required use-case realizations |
| are implemented, as are the interfaces for all classes. Because the phase objective is completeness, complete |
| test-first design is appropriate. Later events should invalidate few, if any, tests. |
| </p> |
| <p> |
| The Inception and Transition phases typically have less focus on design activities for which testing is appropriate. |
| When it is, test-first design is applicable. For example, it could be used with candidate proof-of-concept work in |
| Inception. As with Construction and Elaboration phase testing, it should be aligned with iteration objectives. |
| </p></mainDescription> |
| </org.eclipse.epf.uma:ContentDescription> |