XP/xp/guidances/concepts/test-ideas_catalog.xmi - epf/org.eclipse.epf.libraries - Git at Google

 <?xml version="1.0" encoding="UTF-8"?>
 <org.eclipse.epf.uma:ContentDescription xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:org.eclipse.epf.uma="http://www.eclipse.org/epf/uma/1.0.3/uma.ecore" epf:version="1.0.0" xmi:id="-OrjIrRLW6v_XnqLUQ9GYaQ" name="test-ideas_catalog,1.2384224477983028E-305" guid="-OrjIrRLW6v_XnqLUQ9GYaQ" changeDate="2006-12-01T16:08:47.336-0800" version="1.0.0">
   <mainDescription>&lt;a id=&quot;XE_test-ideas_catalog__concept_of&quot; name=&quot;XE_test-ideas_catalog__concept_of&quot;&gt;&lt;/a&gt;
 &lt;h3&gt;
     &lt;a id=&quot;Introduction&quot; name=&quot;Introduction&quot;&gt;Introduction&lt;/a&gt;
 &lt;/h3&gt;
 &lt;p&gt;
     Much of programming involves taking things you've used over and over before, and then using them yet again in a
     different context. Those things are typically of certain classes-data structures (such as linked lists, hash tables, or
     relational databases) or &lt;a href=&quot;./../../../glossary/glossary.htm#operation&quot; target=&quot;_blank&quot;&gt;&lt;i&gt;operations&lt;/i&gt;&lt;/a&gt;
     (such as searching, sorting, creating temporary files, or popping up a browser window). For example, two customer
     relational databases will have many clichéd characteristics.
 &lt;/p&gt;
 &lt;p&gt;
     The interesting thing about these clichés is that they have clichéd &lt;a href=&quot;./../../../glossary/glossary.htm#fault&quot;
     target=&quot;_blank&quot;&gt;&lt;i&gt;faults&lt;/i&gt;&lt;/a&gt;. People do not invent imaginative new ways to insert something incorrectly into a
     doubly-linked list. They tend to make the same mistakes that they and others have made before. A programmer who pops up
     a browser window might make one of these clichéd mistakes:
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         creates a new window when one that's already open should be reused
     &lt;/li&gt;
     &lt;li&gt;
         fails to make an obscured or minimized browser window visible
     &lt;/li&gt;
     &lt;li&gt;
         uses Internet Explorer when the user has chosen a different default browser
     &lt;/li&gt;
     &lt;li&gt;
         fails to check whether JavaScript is enabled
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;
     Since faults are clichéd, so are the &lt;a href=&quot;./../../../glossary/glossary.htm#test_idea&quot; target=&quot;_blank&quot;&gt;&lt;i&gt;test
     ideas&lt;/i&gt;&lt;/a&gt; that can find them. Put these test ideas in your test-idea catalog so you can reuse them.
 &lt;/p&gt;
 &lt;h3&gt;
     &lt;a id=&quot;HowCatalogsFindFaults&quot; name=&quot;HowCatalogsFindFaults&quot;&gt;How a Test-Ideas Catalog Finds Faults&lt;/a&gt;
 &lt;/h3&gt;
 &lt;p&gt;
     One of the virtues of a catalog is that a single test idea can be useful for finding more than one underlying fault.
     Here's an example of one idea that finds two faults.
 &lt;/p&gt;
 &lt;p&gt;
     The first fault was in a C compiler. This compiler took command-line options like &quot;-table&quot; or &quot;-trace&quot; or &quot;-nolink&quot;.
     The options could be abbreviated to their smallest unique form. For example, &quot;-ta&quot; was as good as &quot;-table&quot;. However,
     &quot;-t&quot; was not allowed, because it was ambiguous: it could mean either &quot;-table&quot; or &quot;-trace&quot;.
 &lt;/p&gt;
 &lt;p&gt;
     Internally, the command-line options were stored in a table like this:
 &lt;/p&gt;
 &lt;div align=&quot;left&quot;&gt;
     &lt;table cellspacing=&quot;0&quot; cellpadding=&quot;2&quot; width=&quot;25%&quot; border=&quot;1&quot;&gt;
         &lt;tbody&gt;
             &lt;tr&gt;
                 &lt;td width=&quot;100%&quot;&gt;
                     -table
                 &lt;/td&gt;
             &lt;/tr&gt;
             &lt;tr&gt;
                 &lt;td width=&quot;100%&quot;&gt;
                     -trace
                 &lt;/td&gt;
             &lt;/tr&gt;
             &lt;tr&gt;
                 &lt;td width=&quot;100%&quot;&gt;
                     -nolink
                 &lt;/td&gt;
             &lt;/tr&gt;
         &lt;/tbody&gt;
     &lt;/table&gt;
     &lt;br /&gt;
 &lt;/div&gt;
 &lt;p&gt;
     When an option was encountered on the command line, it was looked up in the table. It matched if it was the prefix of
     any table entry; that is, &quot;-t&quot; matched &quot;-table&quot;. After one match was found, the rest of the table was searched for
     another match. Another match would be an error, because it would indicate ambiguity.
 &lt;/p&gt;
 &lt;p&gt;
     The code that did the searching looked like this:
 &lt;/p&gt;
 &lt;blockquote&gt;
 &lt;pre&gt;
 for (first=0; first &amp;lt; size; first++) {
     if (matches(entry[first], thing_sought)) {
 /* at least one match */
 for(dup=first+1; dup &amp;lt; size; dup++)
     /* search for another */
     if (matches(entry[dup], thing_sought))
         /* extra match */
         break;  /* error out */
 return first;
     }
 }
 return -1;  /* Not found or ambiguity */
 &lt;/pre&gt;
 &lt;/blockquote&gt;
 &lt;p&gt;
     Do you see the problem? It's fairly subtle.
 &lt;/p&gt;
 &lt;p&gt;
     The problem is the break statement. It's intended to break out of the outermost enclosing loop when a duplicate match
     is found, but it really breaks out of the inner one. That has the same effect as not finding a second match: the index
     of the first match is returned.
 &lt;/p&gt;
 &lt;p&gt;
     Notice that this fault can only be found if the option being sought for matches twice in the table, as &quot;-t&quot; would.
 &lt;/p&gt;
 &lt;p&gt;
     Now let's look at a second, completely different fault.
 &lt;/p&gt;
 &lt;p&gt;
     The code takes a string. It is supposed to replace the last '=' in the string with a '+'. If there is no '=', nothing
     is done. The code uses the standard C library routine &lt;font size=&quot;+0&quot;&gt;strchr&lt;/font&gt; to find the location of '='. Here's
     the code:
 &lt;/p&gt;
 &lt;blockquote&gt;
 &lt;pre&gt;
 ptr = strchr(string, '=');  /* Find last = */
 if (ptr != NULL_CHAR)
     *ptr = '+';
 &lt;/pre&gt;
 &lt;/blockquote&gt;
 &lt;p&gt;
     This problem here is also somewhat subtle.
 &lt;/p&gt;
 &lt;p&gt;
     The function &lt;font size=&quot;+0&quot;&gt;strchr&lt;/font&gt; returns the &lt;i&gt;first&lt;/i&gt; match in the string, not the last. The correct
     function is &lt;font size=&quot;+0&quot;&gt;str&lt;b&gt;&lt;u&gt;r&lt;/u&gt;&lt;/b&gt;chr&lt;/font&gt;. The problem was most likely a typographical error. (Actually,
     the deep underlying problem is that it's definitely unwise to put two functions that differ only by a typo into a
     standard library.)
 &lt;/p&gt;
 &lt;p&gt;
     This fault can only be found when there are two or more equal signs in the input. That is:
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         &quot;a=b&quot; would return the correct result, &quot;a+b&quot;.
     &lt;/li&gt;
     &lt;li&gt;
         &quot;noequals&quot; would return the correct result, &quot;noequals&quot;.
     &lt;/li&gt;
     &lt;li&gt;
         &quot;a=b=c&quot; would incorrectly return &quot;a+b=c&quot;, not the correct &quot;a=b+c&quot;
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;
     What's interesting and useful here is that we have two faults with completely different root causes (typographical
     error, misunderstanding of a C construct) and different manifestations in the code (wrong function called, misuse of
     break statement) that can be found by the &lt;i&gt;same&lt;/i&gt; test idea (search for something that occurs twice).
 &lt;/p&gt;
 &lt;h3&gt;
     &lt;a id=&quot;GoodCatalogs&quot; name=&quot;GoodCatalogs&quot;&gt;A Good Test-Ideas Catalog&lt;/a&gt;
 &lt;/h3&gt;
 &lt;p&gt;
     What makes a good catalog?
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         It contains a small set of test ideas that can find a much larger set of underlying faults.
     &lt;/li&gt;
     &lt;li&gt;
         It's easy to read quickly (skim). You should be able to skip test ideas that are not relevant to your situation.
     &lt;/li&gt;
     &lt;li&gt;
         It contains only test ideas that you will use. For example, someone who doesn't ever deal with Web browsers
         shouldn't have to keep skipping over test ideas for programs that use Web browsers. Someone working on game
         software will want a shorter catalog than someone working on safety-critical software. The game person can afford
         to concentrate only on the test ideas with the highest chance of finding faults.
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;
     Given these rules, it seems best to have more than one catalog. Some data and operations are common to all programming,
     so their test ideas can be put into a catalog that all programmers can use. Others are specific to a particular domain,
     so test ideas for them can be put into a catalog of domain-specific test ideas.
 &lt;/p&gt;
 &lt;p&gt;
     A sample catalog (&lt;a href=&quot;./resources/tstids_short-catalog.pdf&quot; target=&quot;_blank&quot;&gt;tstids_short-catalog.pdf&lt;/a&gt;) (&lt;a
     href=&quot;http://www.adobe.com/products/acrobat/alternate.html&quot;&gt;Get Adobe Reader&lt;/a&gt;), used in the following example, is a
     good one from which to begin. &lt;a href=&quot;../../../examples/extrnlcntrbtns/test/tstatmtch.htm&quot;&gt;Test Ideas for Mixtures of
     ANDs and ORs&lt;/a&gt; provides another example.
 &lt;/p&gt;
 &lt;h3&gt;
     &lt;a id=&quot;UsingACatalogExample&quot; name=&quot;UsingACatalogExample&quot;&gt;An Example of Using a Test-Ideas Catalog&lt;/a&gt;
 &lt;/h3&gt;
 &lt;p&gt;
     Here's how you might use the sample catalog (&lt;a href=&quot;./resources/tstids_short-catalog.pdf&quot;
     target=&quot;_blank&quot;&gt;tstids_short-catalog.pdf&lt;/a&gt;) (&lt;a href=&quot;http://www.adobe.com/products/acrobat/alternate.html&quot;&gt;Get
     Acrobat Reader&lt;/a&gt;). Suppose you're implementing this method:
 &lt;/p&gt;
 &lt;blockquote&gt;
 &lt;pre&gt;
 void applyToCommonFiles(Directory d1,
                 Directory d2,
                 Operation op);
 &lt;/pre&gt;
 &lt;/blockquote&gt;
 &lt;p&gt;
     &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; takes two directories as arguments. When a file in the first directory has
     the same name as a file in the second, &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; performs some operation on that pair
     of files. It descends subdirectories.
 &lt;/p&gt;
 &lt;p&gt;
     The method for using the catalog is to scan through it looking for major headings that match your situation. Consider
     the test ideas under each heading to see if they are relevant, and then write those that are relevant into a &lt;a
     class=&quot;elementLink&quot; href=&quot;./../../../xp/guidances/concepts/test-ideas_list,8.834380241450745E-306.html&quot;
     guid=&quot;8.834380241450745E-306&quot;&gt;Test-Ideas List&lt;/a&gt;.
 &lt;/p&gt;
 &lt;p&gt;
     &lt;b&gt;Note:&lt;/b&gt; This step-by-step description might make using the catalog seem laborious. It takes longer to read about
     creating the checklist than it does to actually create one.
 &lt;/p&gt;
 &lt;p&gt;
     So, in the case of &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt;, you might apply the catalog in the manner described
     throughout the rest of this section.
 &lt;/p&gt;
 &lt;p&gt;
     The first entry is for &lt;b&gt;Any Object&lt;/b&gt;. Could any of the arguments be null pointers? This is a matter of the contract
     between &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; and its callers. The contract could be that the callers will not pass
     in a null pointer. If they do, you can't rely on th expected behavior: &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; could
     perform any action. In such a case, no test is appropriate, since nothing &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt;
     does can be wrong. If, however, &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; is required to check for null pointers, the
     test idea would be useful. Let's assume the latter, which gives us this starting Test-Ideas List:
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         d1 is null (error case)
     &lt;/li&gt;
     &lt;li&gt;
         d2 is null (error case)
     &lt;/li&gt;
     &lt;li&gt;
         op is null (error case)
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;
     The next catalog entry is &lt;b&gt;Strings&lt;/b&gt;. The names of the files are strings, and they're compared to see if they
     match. The idea of testing with the empty string (&quot;&quot;) doesn't seem useful. Presumably some standard string comparison
     routines will be used, and they will handle empty strings correctly.
 &lt;/p&gt;
 &lt;p&gt;
     But wait... If there are strings being compared, what about case? Suppose &lt;font size=&quot;+0&quot;&gt;d1&lt;/font&gt; contains a file
     named &quot;File&quot; and &lt;font size=&quot;+0&quot;&gt;d2&lt;/font&gt; contains a file named &quot;file&quot;. Should those files match? On UNIX, clearly
     not. On Microsoft&amp;reg; Windows&amp;reg;, they almost certainly should. That's another test idea:
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         Files match in the two directories, but the case of the names is different.
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;
     Notice that this test idea didn't come directly from the catalog. However, the catalog drew our attention to a
     particular aspect of the program (file names as strings), and our creativity gave us an additional idea. It's important
     not to use the catalog too narrowly-use it as a brainstorming technique, a way of inspiring new ideas.
 &lt;/p&gt;
 &lt;p&gt;
     The next entry is &lt;b&gt;Collections&lt;/b&gt;. A directory is a collection of files. Many programs that handle collections fail
     on the empty collection. A few that handle the empty collection, or collections with many elements, fail on collections
     with exactly one element. So these ideas are useful:
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         d1 is empty
     &lt;/li&gt;
     &lt;li&gt;
         d2 is empty
     &lt;/li&gt;
     &lt;li&gt;
         d1 has exactly one file
     &lt;/li&gt;
     &lt;li&gt;
         d2 has exactly one file
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;
     The next idea is to use a collection of the maximum possible size. This is useful because programs like &lt;font
     size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; are often tested with trivial little directories. Then some user comes along and
     applies them to two huge directory trees with thousands of files in them, only to discover that the program is
     grotesquely memory inefficient and can't handle that realistic case.
 &lt;/p&gt;
 &lt;p&gt;
     Now, testing the absolute maximum size for a directory is not important; it only needs to be as large as a user might
     try. However, at the very least, there should be &lt;i&gt;some&lt;/i&gt; test with more than three files in a directory:
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         d1 contains very many files
     &lt;/li&gt;
     &lt;li&gt;
         d2 contains very many files
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;
     The final test idea (duplicate elements) doesn't apply to directories of files. That is, if you have a directory with
     two files that have the same name, you have a problem independent of &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt;-your
     file system is corrupt.
 &lt;/p&gt;
 &lt;p&gt;
     The next catalog entry is &lt;b&gt;Searching&lt;/b&gt;. Those ideas can be translated into &lt;font
     size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; terms like this:
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         d1 and d2 have no files in common (all the names are different)
     &lt;/li&gt;
     &lt;li&gt;
         d1 and d2 have exactly one file in common (it's alphabetically the last element in the directory)
     &lt;/li&gt;
     &lt;li&gt;
         d1 and d2 have more than one file in common
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;
     The final test idea checks whether &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; terminates too soon. Does it return as
     soon as it finds the first match? The parenthetical remark in the test idea before that assumes that the program will
     fetch the list of files in a directory using some library routine that returns them, sorted alphabetically. If not, it
     might be better to find out what the last one really is (the most recently created?) and make that be the match. Before
     you devote a lot of time to finding out how files are ordered, though, ask yourself how likely it is that putting the
     matching element last will make finding defects easier. Putting an element last in a collection is more useful if the
     code explicitly steps through the collection using an index. If it's using an iterator, it's extremely unlikely that
     the order matters.
 &lt;/p&gt;
 &lt;p&gt;
     Let's look at one more entry in the sample catalog. The &lt;b&gt;Linked structures&lt;/b&gt; entry reminds us that we're comparing
     directory &lt;i&gt;trees&lt;/i&gt;, not just flat collections of files. It would be sad if &lt;font
     size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; worked only in the top-level directories, but not in the lower-level ones. Deciding
     how to test whether &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; works in lower-level directories forces us to confront
     the incompleteness of its description.
 &lt;/p&gt;
 &lt;p&gt;
     First, when does &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; descend into subdirectories? If the directory structure
     looks like this
 &lt;/p&gt;
 &lt;p align=&quot;center&quot;&gt;
     &lt;img height=&quot;162&quot; alt=&quot;&quot; src=&quot;resources/tstidsctl-img1.gif&quot; width=&quot;334&quot; /&gt;
 &lt;/p&gt;
 &lt;p class=&quot;picturetext&quot;&gt;
     Figure 1: A directory structure
 &lt;/p&gt;
 &lt;p&gt;
     does &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; descend into &lt;font size=&quot;+0&quot;&gt;Cdir&lt;/font&gt;? That doesn't seem to make
     sense. There can be no match with anything in the other directory tree. In fact, it seems as if files in subdirectories
     can only match if the subdirectory names match. That is, suppose we have this directory structure:
 &lt;/p&gt;
 &lt;p align=&quot;center&quot;&gt;
     &lt;img height=&quot;165&quot; alt=&quot;&quot; src=&quot;resources/tstidsctl-img2.gif&quot; width=&quot;334&quot; /&gt;
 &lt;/p&gt;
 &lt;p class=&quot;picturetext&quot;&gt;
     Figure 2: A second directory structure
 &lt;/p&gt;
 &lt;p&gt;
     The files named &quot;File&quot; don't match because they're in different subdirectories The subdirectories should be descended
     only if they have the same name in both &lt;font size=&quot;+0&quot;&gt;d1&lt;/font&gt; and &lt;font size=&quot;+0&quot;&gt;d2&lt;/font&gt;. That leads to these
     test ideas:
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         some subdirectory in d1 is not found in d2 (no descent)
     &lt;/li&gt;
     &lt;li&gt;
         some subdirectory in d2 is not found in d1 (no descent)
     &lt;/li&gt;
     &lt;li&gt;
         some subdirectory appears in both d1 and d2 (descend)
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;
     But that raises other questions. Should the operation (&lt;font size=&quot;+0&quot;&gt;op&lt;/font&gt;) be applied to matching subdirectories
     or just to matching files? If it's applied to the subdirectories, should it be applied before the descent or afterward?
     That makes a difference if, for example, the operation deletes the matching file or directory. For that matter,
     &lt;i&gt;should&lt;/i&gt; the operation be allowed to modify the directory structure? And more specifically: what's the correct
     behavior of &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; if it does? (This is the same issue that comes up with
     iterators.)
 &lt;/p&gt;
 &lt;p&gt;
     These sorts of questions typically arise when you carefully read a method's description of creating test ideas. But
     let's leave them aside for now. Whatever the answers are, there will have to be test ideas for them-test ideas that
     check whether the code correctly implements the answers.
 &lt;/p&gt;
 &lt;p&gt;
     Let's return to the catalog. We still haven't considered all of its test ideas. The first one-empty (nothing in
     structure)-asks for an empty directory. We've already got that from the &lt;b&gt;Collections&lt;/b&gt; entry. We've also got the
     &lt;b&gt;minimal non-empty&lt;/b&gt; structure, which is a directory with a single element. This sort of redundancy is not
     uncommon, but it's easy to ignore.
 &lt;/p&gt;
 &lt;p&gt;
     What about &lt;b&gt;a circular structure&lt;/b&gt;? Directory structures can't be circular-a directory can't be within one of its
     descendants or within itself... or can it? What about shortcuts (on Windows) or symbolic links (on UNIX)? If there's a
     shortcut in &lt;font size=&quot;+0&quot;&gt;d1&lt;/font&gt;'s directory tree that points back to &lt;font size=&quot;+0&quot;&gt;d1&lt;/font&gt;, should &lt;font
     size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; keep descending forever? The answer could lead to one or more new test ideas:
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         d1 is circular because of shortcuts or symbolic links
     &lt;/li&gt;
     &lt;li&gt;
         d2 is circular because of shortcuts or symbolic links
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;
     Depending on the correct behavior, there may be more test ideas than that.
 &lt;/p&gt;
 &lt;p&gt;
     Finally, what about &lt;b&gt;depth greater than one&lt;/b&gt;? Earlier test ideas will ensure that we test descending into one
     level of subdirectory, but we should check that &lt;font size=&quot;+0&quot;&gt;applyToCommonFiles&lt;/font&gt; keeps descending:
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         descends through several levels (&amp;gt;1) of d1's subdirectories
     &lt;/li&gt;
     &lt;li&gt;
         descends through several levels (&amp;gt;1) of d2's subdirectories
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;h3&gt;
     &lt;a id=&quot;CreatingYourOwnCatalogs&quot; name=&quot;CreatingYourOwnCatalogs&quot;&gt;Creating and Maintaining Your Own Test-Ideas Catalog&lt;/a&gt;
 &lt;/h3&gt;
 &lt;p&gt;
     As mentioned previously, the generic catalog won't contain all of the test ideas you need. But &lt;a
     href=&quot;./../../../glossary/glossary.htm#domain&quot; target=&quot;_blank&quot;&gt;&lt;i&gt;domain&lt;/i&gt;&lt;/a&gt;-specific catalogs haven't been
     published outside of the companies that created them. If you want them, you'll need to build them. Here's some advice.
 &lt;/p&gt;
 &lt;ul&gt;
     &lt;li&gt;
         Do not fill a catalog with your speculations about what ideas would be good for finding faults. Remember that each
         test idea you put in the catalog costs time and money:
         &lt;ul&gt;
             &lt;li&gt;
                 your time to maintain the catalog
             &lt;/li&gt;
             &lt;li&gt;
                 other programmers' time to think about the test idea
             &lt;/li&gt;
             &lt;li&gt;
                 possibly other programmers' time to implement a test
             &lt;/li&gt;
         &lt;/ul&gt;
         &lt;br /&gt;
          Add only ideas that have a demonstrated track record. You should be able to point to at least one actual fault
         that the test idea would have caught. Ideally, the fault should be one that was missed by other testing; that is,
         one that was reported from the field. One good way to build catalogs is to browse through your company's bug
         database and ask questions about how each fault could have been detected earlier.
     &lt;/li&gt;
     &lt;li style=&quot;LIST-STYLE-TYPE: none&quot;&gt;
         &lt;br /&gt;
         &lt;br /&gt;
     &lt;/li&gt;
     &lt;li&gt;
         It's unlikely to work if creating and maintaining a Test-Ideas Catalog is something you do in your spare time.
         You'll need time specifically allocated to this task, just like for any other important one. We recommend you
         create and maintain your Test-Ideas Catalog during &lt;a href=&quot;wfs_imptstast.htm&quot;&gt;Workflow Detail: Improve Test
         Assets&lt;/a&gt;.
     &lt;/li&gt;
 &lt;/ul&gt;
 &lt;br /&gt;
 &lt;br /&gt;</mainDescription>
 </org.eclipse.epf.uma:ContentDescription>
	<?xml version="1.0" encoding="UTF-8"?>
	<org.eclipse.epf.uma:ContentDescription xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:org.eclipse.epf.uma="http://www.eclipse.org/epf/uma/1.0.3/uma.ecore" epf:version="1.0.0" xmi:id="-OrjIrRLW6v_XnqLUQ9GYaQ" name="test-ideas_catalog,1.2384224477983028E-305" guid="-OrjIrRLW6v_XnqLUQ9GYaQ" changeDate="2006-12-01T16:08:47.336-0800" version="1.0.0">
	<mainDescription><a id="XE_test-ideas_catalog__concept_of" name="XE_test-ideas_catalog__concept_of"></a>
	<h3>
	<a id="Introduction" name="Introduction">Introduction</a>
	</h3>
	<p>
	Much of programming involves taking things you've used over and over before, and then using them yet again in a
	different context. Those things are typically of certain classes-data structures (such as linked lists, hash tables, or
	relational databases) or <a href="./../../../glossary/glossary.htm#operation" target="_blank"><i>operations</i></a>
	(such as searching, sorting, creating temporary files, or popping up a browser window). For example, two customer
	relational databases will have many clichéd characteristics.
	</p>
	<p>
	The interesting thing about these clichés is that they have clichéd <a href="./../../../glossary/glossary.htm#fault"
	target="_blank"><i>faults</i></a>. People do not invent imaginative new ways to insert something incorrectly into a
	doubly-linked list. They tend to make the same mistakes that they and others have made before. A programmer who pops up
	a browser window might make one of these clichéd mistakes:
	</p>
	<ul>
	<li>
	creates a new window when one that's already open should be reused
	</li>
	<li>
	fails to make an obscured or minimized browser window visible
	</li>
	<li>
	uses Internet Explorer when the user has chosen a different default browser
	</li>
	<li>
	fails to check whether JavaScript is enabled
	</li>
	</ul>
	<p>
	Since faults are clichéd, so are the <a href="./../../../glossary/glossary.htm#test_idea" target="_blank"><i>test
	ideas</i></a> that can find them. Put these test ideas in your test-idea catalog so you can reuse them.
	</p>
	<h3>
	<a id="HowCatalogsFindFaults" name="HowCatalogsFindFaults">How a Test-Ideas Catalog Finds Faults</a>
	</h3>
	<p>
	One of the virtues of a catalog is that a single test idea can be useful for finding more than one underlying fault.
	Here's an example of one idea that finds two faults.
	</p>
	<p>
	The first fault was in a C compiler. This compiler took command-line options like "-table" or "-trace" or "-nolink".
	The options could be abbreviated to their smallest unique form. For example, "-ta" was as good as "-table". However,
	"-t" was not allowed, because it was ambiguous: it could mean either "-table" or "-trace".
	</p>
	<p>
	Internally, the command-line options were stored in a table like this:
	</p>
	<div align="left">
	<table cellspacing="0" cellpadding="2" width="25%" border="1">
	<tbody>
	<tr>
	<td width="100%">
	-table
	</td>
	</tr>
	<tr>
	<td width="100%">
	-trace
	</td>
	</tr>
	<tr>
	<td width="100%">
	-nolink
	</td>
	</tr>
	</tbody>
	</table>
	<br />
	</div>
	<p>
	When an option was encountered on the command line, it was looked up in the table. It matched if it was the prefix of
	any table entry; that is, "-t" matched "-table". After one match was found, the rest of the table was searched for
	another match. Another match would be an error, because it would indicate ambiguity.
	</p>
	<p>
	The code that did the searching looked like this:
	</p>
	<blockquote>
	<pre>
	for (first=0; first &lt; size; first++) {
	if (matches(entry[first], thing_sought)) {
	/* at least one match */
	for(dup=first+1; dup &lt; size; dup++)
	/* search for another */
	if (matches(entry[dup], thing_sought))
	/* extra match */
	break; /* error out */
	return first;
	}
	}
	return -1; /* Not found or ambiguity */
	</pre>
	</blockquote>
	<p>
	Do you see the problem? It's fairly subtle.
	</p>
	<p>
	The problem is the break statement. It's intended to break out of the outermost enclosing loop when a duplicate match
	is found, but it really breaks out of the inner one. That has the same effect as not finding a second match: the index
	of the first match is returned.
	</p>
	<p>
	Notice that this fault can only be found if the option being sought for matches twice in the table, as "-t" would.
	</p>
	<p>
	Now let's look at a second, completely different fault.
	</p>
	<p>
	The code takes a string. It is supposed to replace the last '=' in the string with a '+'. If there is no '=', nothing
	is done. The code uses the standard C library routine <font size="+0">strchr</font> to find the location of '='. Here's
	the code:
	</p>
	<blockquote>
	<pre>
	ptr = strchr(string, '='); /* Find last = */
	if (ptr != NULL_CHAR)
	*ptr = '+';
	</pre>
	</blockquote>
	<p>
	This problem here is also somewhat subtle.
	</p>
	<p>
	The function <font size="+0">strchr</font> returns the <i>first</i> match in the string, not the last. The correct
	function is <font size="+0">str<b><u>r</u></b>chr</font>. The problem was most likely a typographical error. (Actually,
	the deep underlying problem is that it's definitely unwise to put two functions that differ only by a typo into a
	standard library.)
	</p>
	<p>
	This fault can only be found when there are two or more equal signs in the input. That is:
	</p>
	<ul>
	<li>
	"a=b" would return the correct result, "a+b".
	</li>
	<li>
	"noequals" would return the correct result, "noequals".
	</li>
	<li>
	"a=b=c" would incorrectly return "a+b=c", not the correct "a=b+c"
	</li>
	</ul>
	<p>
	What's interesting and useful here is that we have two faults with completely different root causes (typographical
	error, misunderstanding of a C construct) and different manifestations in the code (wrong function called, misuse of
	break statement) that can be found by the <i>same</i> test idea (search for something that occurs twice).
	</p>
	<h3>
	<a id="GoodCatalogs" name="GoodCatalogs">A Good Test-Ideas Catalog</a>
	</h3>
	<p>
	What makes a good catalog?
	</p>
	<ul>
	<li>
	It contains a small set of test ideas that can find a much larger set of underlying faults.
	</li>
	<li>
	It's easy to read quickly (skim). You should be able to skip test ideas that are not relevant to your situation.
	</li>
	<li>
	It contains only test ideas that you will use. For example, someone who doesn't ever deal with Web browsers
	shouldn't have to keep skipping over test ideas for programs that use Web browsers. Someone working on game
	software will want a shorter catalog than someone working on safety-critical software. The game person can afford
	to concentrate only on the test ideas with the highest chance of finding faults.
	</li>
	</ul>
	<p>
	Given these rules, it seems best to have more than one catalog. Some data and operations are common to all programming,
	so their test ideas can be put into a catalog that all programmers can use. Others are specific to a particular domain,
	so test ideas for them can be put into a catalog of domain-specific test ideas.
	</p>
	<p>
	A sample catalog (<a href="./resources/tstids_short-catalog.pdf" target="_blank">tstids_short-catalog.pdf</a>) (<a
	href="http://www.adobe.com/products/acrobat/alternate.html">Get Adobe Reader</a>), used in the following example, is a
	good one from which to begin. <a href="../../../examples/extrnlcntrbtns/test/tstatmtch.htm">Test Ideas for Mixtures of
	ANDs and ORs</a> provides another example.
	</p>
	<h3>
	<a id="UsingACatalogExample" name="UsingACatalogExample">An Example of Using a Test-Ideas Catalog</a>
	</h3>
	<p>
	Here's how you might use the sample catalog (<a href="./resources/tstids_short-catalog.pdf"
	target="_blank">tstids_short-catalog.pdf</a>) (<a href="http://www.adobe.com/products/acrobat/alternate.html">Get
	Acrobat Reader</a>). Suppose you're implementing this method:
	</p>
	<blockquote>
	<pre>
	void applyToCommonFiles(Directory d1,
	Directory d2,
	Operation op);
	</pre>
	</blockquote>
	<p>
	<font size="+0">applyToCommonFiles</font> takes two directories as arguments. When a file in the first directory has
	the same name as a file in the second, <font size="+0">applyToCommonFiles</font> performs some operation on that pair
	of files. It descends subdirectories.
	</p>
	<p>
	The method for using the catalog is to scan through it looking for major headings that match your situation. Consider
	the test ideas under each heading to see if they are relevant, and then write those that are relevant into a <a
	class="elementLink" href="./../../../xp/guidances/concepts/test-ideas_list,8.834380241450745E-306.html"
	guid="8.834380241450745E-306">Test-Ideas List</a>.
	</p>
	<p>
	<b>Note:</b> This step-by-step description might make using the catalog seem laborious. It takes longer to read about
	creating the checklist than it does to actually create one.
	</p>
	<p>
	So, in the case of <font size="+0">applyToCommonFiles</font>, you might apply the catalog in the manner described
	throughout the rest of this section.
	</p>
	<p>
	The first entry is for <b>Any Object</b>. Could any of the arguments be null pointers? This is a matter of the contract
	between <font size="+0">applyToCommonFiles</font> and its callers. The contract could be that the callers will not pass
	in a null pointer. If they do, you can't rely on th expected behavior: <font size="+0">applyToCommonFiles</font> could
	perform any action. In such a case, no test is appropriate, since nothing <font size="+0">applyToCommonFiles</font>
	does can be wrong. If, however, <font size="+0">applyToCommonFiles</font> is required to check for null pointers, the
	test idea would be useful. Let's assume the latter, which gives us this starting Test-Ideas List:
	</p>
	<ul>
	<li>
	d1 is null (error case)
	</li>
	<li>
	d2 is null (error case)
	</li>
	<li>
	op is null (error case)
	</li>
	</ul>
	<p>
	The next catalog entry is <b>Strings</b>. The names of the files are strings, and they're compared to see if they
	match. The idea of testing with the empty string ("") doesn't seem useful. Presumably some standard string comparison
	routines will be used, and they will handle empty strings correctly.
	</p>
	<p>
	But wait... If there are strings being compared, what about case? Suppose <font size="+0">d1</font> contains a file
	named "File" and <font size="+0">d2</font> contains a file named "file". Should those files match? On UNIX, clearly
	not. On Microsoft&reg; Windows&reg;, they almost certainly should. That's another test idea:
	</p>
	<ul>
	<li>
	Files match in the two directories, but the case of the names is different.
	</li>
	</ul>
	<p>
	Notice that this test idea didn't come directly from the catalog. However, the catalog drew our attention to a
	particular aspect of the program (file names as strings), and our creativity gave us an additional idea. It's important
	not to use the catalog too narrowly-use it as a brainstorming technique, a way of inspiring new ideas.
	</p>
	<p>
	The next entry is <b>Collections</b>. A directory is a collection of files. Many programs that handle collections fail
	on the empty collection. A few that handle the empty collection, or collections with many elements, fail on collections
	with exactly one element. So these ideas are useful:
	</p>
	<ul>
	<li>
	d1 is empty
	</li>
	<li>
	d2 is empty
	</li>
	<li>
	d1 has exactly one file
	</li>
	<li>
	d2 has exactly one file
	</li>
	</ul>
	<p>
	The next idea is to use a collection of the maximum possible size. This is useful because programs like <font
	size="+0">applyToCommonFiles</font> are often tested with trivial little directories. Then some user comes along and
	applies them to two huge directory trees with thousands of files in them, only to discover that the program is
	grotesquely memory inefficient and can't handle that realistic case.
	</p>
	<p>
	Now, testing the absolute maximum size for a directory is not important; it only needs to be as large as a user might
	try. However, at the very least, there should be <i>some</i> test with more than three files in a directory:
	</p>
	<ul>
	<li>
	d1 contains very many files
	</li>
	<li>
	d2 contains very many files
	</li>
	</ul>
	<p>
	The final test idea (duplicate elements) doesn't apply to directories of files. That is, if you have a directory with
	two files that have the same name, you have a problem independent of <font size="+0">applyToCommonFiles</font>-your
	file system is corrupt.
	</p>
	<p>
	The next catalog entry is <b>Searching</b>. Those ideas can be translated into <font
	size="+0">applyToCommonFiles</font> terms like this:
	</p>
	<ul>
	<li>
	d1 and d2 have no files in common (all the names are different)
	</li>
	<li>
	d1 and d2 have exactly one file in common (it's alphabetically the last element in the directory)
	</li>
	<li>
	d1 and d2 have more than one file in common
	</li>
	</ul>
	<p>
	The final test idea checks whether <font size="+0">applyToCommonFiles</font> terminates too soon. Does it return as
	soon as it finds the first match? The parenthetical remark in the test idea before that assumes that the program will
	fetch the list of files in a directory using some library routine that returns them, sorted alphabetically. If not, it
	might be better to find out what the last one really is (the most recently created?) and make that be the match. Before
	you devote a lot of time to finding out how files are ordered, though, ask yourself how likely it is that putting the
	matching element last will make finding defects easier. Putting an element last in a collection is more useful if the
	code explicitly steps through the collection using an index. If it's using an iterator, it's extremely unlikely that
	the order matters.
	</p>
	<p>
	Let's look at one more entry in the sample catalog. The <b>Linked structures</b> entry reminds us that we're comparing
	directory <i>trees</i>, not just flat collections of files. It would be sad if <font
	size="+0">applyToCommonFiles</font> worked only in the top-level directories, but not in the lower-level ones. Deciding
	how to test whether <font size="+0">applyToCommonFiles</font> works in lower-level directories forces us to confront
	the incompleteness of its description.
	</p>
	<p>
	First, when does <font size="+0">applyToCommonFiles</font> descend into subdirectories? If the directory structure
	looks like this
	</p>
	<p align="center">
	<img height="162" alt="" src="resources/tstidsctl-img1.gif" width="334" />
	</p>
	<p class="picturetext">
	Figure 1: A directory structure
	</p>
	<p>
	does <font size="+0">applyToCommonFiles</font> descend into <font size="+0">Cdir</font>? That doesn't seem to make
	sense. There can be no match with anything in the other directory tree. In fact, it seems as if files in subdirectories
	can only match if the subdirectory names match. That is, suppose we have this directory structure:
	</p>
	<p align="center">
	<img height="165" alt="" src="resources/tstidsctl-img2.gif" width="334" />
	</p>
	<p class="picturetext">
	Figure 2: A second directory structure
	</p>
	<p>
	The files named "File" don't match because they're in different subdirectories The subdirectories should be descended
	only if they have the same name in both <font size="+0">d1</font> and <font size="+0">d2</font>. That leads to these
	test ideas:
	</p>
	<ul>
	<li>
	some subdirectory in d1 is not found in d2 (no descent)
	</li>
	<li>
	some subdirectory in d2 is not found in d1 (no descent)
	</li>
	<li>
	some subdirectory appears in both d1 and d2 (descend)
	</li>
	</ul>
	<p>
	But that raises other questions. Should the operation (<font size="+0">op</font>) be applied to matching subdirectories
	or just to matching files? If it's applied to the subdirectories, should it be applied before the descent or afterward?
	That makes a difference if, for example, the operation deletes the matching file or directory. For that matter,
	<i>should</i> the operation be allowed to modify the directory structure? And more specifically: what's the correct
	behavior of <font size="+0">applyToCommonFiles</font> if it does? (This is the same issue that comes up with
	iterators.)
	</p>
	<p>
	These sorts of questions typically arise when you carefully read a method's description of creating test ideas. But
	let's leave them aside for now. Whatever the answers are, there will have to be test ideas for them-test ideas that
	check whether the code correctly implements the answers.
	</p>
	<p>
	Let's return to the catalog. We still haven't considered all of its test ideas. The first one-empty (nothing in
	structure)-asks for an empty directory. We've already got that from the <b>Collections</b> entry. We've also got the
	<b>minimal non-empty</b> structure, which is a directory with a single element. This sort of redundancy is not
	uncommon, but it's easy to ignore.
	</p>
	<p>
	What about <b>a circular structure</b>? Directory structures can't be circular-a directory can't be within one of its
	descendants or within itself... or can it? What about shortcuts (on Windows) or symbolic links (on UNIX)? If there's a
	shortcut in <font size="+0">d1</font>'s directory tree that points back to <font size="+0">d1</font>, should <font
	size="+0">applyToCommonFiles</font> keep descending forever? The answer could lead to one or more new test ideas:
	</p>
	<ul>
	<li>
	d1 is circular because of shortcuts or symbolic links
	</li>
	<li>
	d2 is circular because of shortcuts or symbolic links
	</li>
	</ul>
	<p>
	Depending on the correct behavior, there may be more test ideas than that.
	</p>
	<p>
	Finally, what about <b>depth greater than one</b>? Earlier test ideas will ensure that we test descending into one
	level of subdirectory, but we should check that <font size="+0">applyToCommonFiles</font> keeps descending:
	</p>
	<ul>
	<li>
	descends through several levels (&gt;1) of d1's subdirectories
	</li>
	<li>
	descends through several levels (&gt;1) of d2's subdirectories
	</li>
	</ul>
	<h3>
	<a id="CreatingYourOwnCatalogs" name="CreatingYourOwnCatalogs">Creating and Maintaining Your Own Test-Ideas Catalog</a>
	</h3>
	<p>
	As mentioned previously, the generic catalog won't contain all of the test ideas you need. But <a
	href="./../../../glossary/glossary.htm#domain" target="_blank"><i>domain</i></a>-specific catalogs haven't been
	published outside of the companies that created them. If you want them, you'll need to build them. Here's some advice.
	</p>
	<ul>
	<li>
	Do not fill a catalog with your speculations about what ideas would be good for finding faults. Remember that each
	test idea you put in the catalog costs time and money:
	<ul>
	<li>
	your time to maintain the catalog
	</li>
	<li>
	other programmers' time to think about the test idea
	</li>
	<li>
	possibly other programmers' time to implement a test
	</li>
	</ul>
	<br />
	Add only ideas that have a demonstrated track record. You should be able to point to at least one actual fault
	that the test idea would have caught. Ideally, the fault should be one that was missed by other testing; that is,
	one that was reported from the field. One good way to build catalogs is to browse through your company's bug
	database and ask questions about how each fault could have been detected earlier.
	</li>
	<li style="LIST-STYLE-TYPE: none">
	<br />
	<br />
	</li>
	<li>
	It's unlikely to work if creating and maintaining a Test-Ideas Catalog is something you do in your spare time.
	You'll need time specifically allocated to this task, just like for any other important one. We recommend you
	create and maintain your Test-Ideas Catalog during <a href="wfs_imptstast.htm">Workflow Detail: Improve Test
	Assets</a>.
	</li>
	</ul>
	<br />
	<br /></mainDescription>
	</org.eclipse.epf.uma:ContentDescription>