<?xml version="1.0" encoding="UTF-8"?>
<org.eclipse.epf.uma:ContentDescription xmi:version="2.0"
    xmlns:xmi="http://www.omg.org/XMI" xmlns:org.eclipse.epf.uma="http://www.eclipse.org/epf/uma/1.0.5/uma.ecore"
    xmlns:epf="http://www.eclipse.org/epf" epf:version="1.5.0" xmi:id="-ejZeo5A3u76_1K4bqoFH6g"
    name=",_J-T2kG6ZEdueuJFacHVsEA" guid="-ejZeo5A3u76_1K4bqoFH6g" authors="Pierre Berlandier"
    changeDate="2008-11-20T21:16:53.141-0800" version="1.0.0">
  <mainDescription>&lt;h3>&#xD;
    Abstract&#xD;
&lt;/h3>&#xD;
&lt;p>&#xD;
    Maintaining an adequate level of quality assurance for applications with a high rate of business rules turnover&#xD;
    presents a challenge for many IT departments: the test cases require constant update from business experts to adjust&#xD;
    the expected results to the rule changes. Consequently, only a limited number of test cases can usually be maintained.&#xD;
    We propose in this paper a simple but efficient approach, called delta testing, which by-passes the need for&#xD;
    maintaining expected results, using instead the set of rule currently in production as the reference for testing the&#xD;
    new set of rules. This approach allows the development of more test cases with less involvement from the business&#xD;
    experts, thus providing a more thorough and less expensive QA strategy.&#xD;
&lt;/p>&#xD;
&lt;h3>&#xD;
    &lt;a>Introduction&lt;/a>&#xD;
&lt;/h3>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The process of testing a conventional piece of software is usually implemented by first elaborating a test plan, which&#xD;
    leads to the design of various test suites (1), then running these test suites on a regular basis to ensure the&#xD;
    correctness and stability of the system through its development and maintenance.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    A test case is generally composed of pair of data sets: the input data and the corresponding (expected) output data.&#xD;
    Executing a test case consists in submitting the input data to the system and comparing the &lt;i&#xD;
    style=&quot;mso-bidi-font-style: normal&quot;>actual&lt;/i> output to the &lt;i style=&quot;mso-bidi-font-style: normal&quot;>expected&lt;/i>&#xD;
    output: if they match, the test case execution is considered successful.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The initial investment required in creating the test cases, especially the part about manually devising the expected&#xD;
    output values is usually non-negligible.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    However, this investment gets quickly amortized since the same test suite executions are repeated over time for&#xD;
    regression testing purpose. Also and more importantly, the expected values of a test case do not change, unless the&#xD;
    system requirements themselves are changing, which should rarely occur in a conventional system.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    This assumption does not hold true for business rule based systems. Indeed, the essence of business rules engines and&#xD;
    the reason why they have been brought into the enterprise software picture is to provide a quick turnaround for&#xD;
    frequent business requirement changes and adjustments. Still, the ability to quickly implement and deploy incremental&#xD;
    changes that comes with the business rules paradigm does not exonerate us from applying a stringent unit-testing and&#xD;
    quality assurance process.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    With business rules based systems, the problem that we face is that we need to preserve the rapid implementation and&#xD;
    deployment of changes while still running the set of test cases necessary to ensure the system correctness. If we&#xD;
    decide to stick with the standard approach of comparing expected versus actual results, we need to review and update&#xD;
    the expected values of the test cases whenever the requirements are updated and a new rule set is deployed. As we&#xD;
    mentioned before, devising the expected results of test cases is a step which requires a significant time investment&#xD;
    from a group of precious business Subject Matter Experts (SME).&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    These experts are often a scarce resource, which can hardly, if at all, be supplemented by external contractors. It is&#xD;
    thus essential for the organization IT group to find a way to partly automate the testing of incremental changes in a&#xD;
    rule based application. We present here a testing&lt;a id=&quot;XE_changing_the_rule_of_testing__guideline&quot;&#xD;
    name=&quot;XE_changing_the_rule_of_testing__guideline&quot;>&lt;/a> approach which does not replace, but complements the use of&#xD;
    expected results. It is based on the comparison of the output from the currently tested and deployed rule set (referred&#xD;
    to as the &lt;i style=&quot;mso-bidi-font-style: normal&quot;>champion&lt;/i> rule set) with the output from the updated rule set&#xD;
    currently being tested (referred to as the &lt;i style=&quot;mso-bidi-font-style: normal&quot;>challenger&lt;/i> rule set).&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The concept of pass/fail is based on an equivalence function which semantics is specific to a test suite and which&#xD;
    determines, for a given test case, if the difference between the output from the champion and the challenger rule set&#xD;
    is acceptable or not.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    This incremental difference (or &lt;i style=&quot;mso-bidi-font-style: normal&quot;>delta&lt;/i>) testing approach is more declarative&#xD;
    and local than the traditional one; the question we have to ask ourselves while designing the test cases is now &lt;i&#xD;
    style=&quot;mso-bidi-font-style: normal&quot;>&quot;what changes from the previous version should we expect in the output&quot;&lt;/i> rather&#xD;
    than the traditional &lt;i style=&quot;mso-bidi-font-style: normal&quot;>&quot;what output should we expect&quot;&lt;/i>. The main benefit is&#xD;
    that testing does not depend on the SME resources to define and update the test cases. Also, the approach allows the&#xD;
    quick definition and execution of large sets of tests cases, making quality assurance for rule sets more realistic and&#xD;
    thorough.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    In the following sections, we present in more details the concept, the benefits and the implementation of the&#xD;
    delta-testing approach for business rules.&#xD;
&lt;/p>&#xD;
&lt;h3>&#xD;
    &lt;a>Application in Mortgage Lending&lt;/a>&#xD;
&lt;/h3>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    We selected mortgage underwriting (and pre-qualification) as a supporting application domain example for our&#xD;
    presentation. Mortgage underwriting applications are especially relevant to delta-testing because changes to the&#xD;
    business rules are frequent and incremental. This is particularly true in the sub-prime lending area, where&#xD;
    underwriting guidelines changes often come out weekly, to adjust to ever new and creative lending options and also&#xD;
    heavy competition among lenders.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    Some example of data involved in mortgage underwriting is the type of property (e.g. Condominium, Single Family&#xD;
    Residence, Manufactured Homes), the FICO score of the applicant,&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    Extensive information on the domain (data dictionary, XML DTDs) can be found at &lt;a href=&quot;http://www.mismo.org/&quot;>&lt;font&#xD;
    color=&quot;#005DA0&quot;>http://www.mismo.org&lt;/font>&lt;/a>, the site for the Mortgage Industry Standards Maintenance Organization,&#xD;
    which maintains electronic commerce standards for the mortgage industry.&#xD;
&lt;/p>&#xD;
&lt;h3>&#xD;
    &lt;a>Delta Testing&lt;/a>&#xD;
&lt;/h3>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The data flow of a conventional testing framework is presented on &lt;span&#xD;
    style=&quot;mso-field-code: ' REF _Ref114287457 h '&quot;>Figure &lt;span style=&quot;mso-no-proof: yes&quot;>1&lt;/span>&lt;/span>. The framework&#xD;
    involves a single rule engine which runs the tested rule set and two sets of user data input: the test cases input data&#xD;
    and the expected output data.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    &amp;nbsp;&lt;img height=&quot;304&quot; alt=&quot;&quot; src=&quot;resources/conventional_testing.gif&quot; width=&quot;592&quot; />&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoCaption&quot; style=&quot;MARGIN: 6pt 0cm 6pt 11.35pt; TEXT-ALIGN: center&quot; align=&quot;center&quot;>&#xD;
    &lt;a>&lt;strong>Figure&lt;/strong>&lt;/a> &lt;strong>&lt;span style=&quot;mso-bookmark: _Ref114287457&quot;>&lt;span&#xD;
    style=&quot;mso-no-proof: yes&quot;>1&lt;/span>&lt;/span>: Conventional Testing Framework&lt;/strong>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    By contrast, the delta testing framework involves two rule engines, one for the champion rule set and the other for the&#xD;
    challenger rule set, and a single set of user data input, which is the test cases data input.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    &amp;nbsp;&lt;img height=&quot;440&quot; alt=&quot;&quot; src=&quot;resources/delta_testing.gif&quot; width=&quot;600&quot; />&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoCaption&quot; style=&quot;MARGIN: 6pt 0cm 6pt 11.35pt; TEXT-ALIGN: center&quot; align=&quot;center&quot;>&#xD;
    &lt;strong>Figure &lt;span style=&quot;mso-no-proof: yes&quot;>2&lt;/span>: Delta Testing Framework&lt;/strong>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    While the response comparison process in a conventional testing framework is always based on strict equality, delta&#xD;
    testing needs to use a more flexible notion of equivalence. Using equality leads to &lt;i&#xD;
    style=&quot;mso-bidi-font-style: normal&quot;>qualitative&lt;/i> delta testing. Using a custom equivalence function leads to &lt;i&#xD;
    style=&quot;mso-bidi-font-style: normal&quot;>quantitative&lt;/i> delta testing. These two approaches and their specific use are&#xD;
    detailed in the following sections.&#xD;
&lt;/p>&#xD;
&lt;h4>&#xD;
    &lt;a>Qualitative Delta Testing&lt;/a>&#xD;
&lt;/h4>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The simplest way to exploit delta-testing is to test for strict equality between all the output values from the&#xD;
    champion and challenger rule set. The qualitative indication is whether or not the changes implemented by the&#xD;
    challenger rule set are affecting the outcome of the test case execution. As a result, we get a partition of the test&#xD;
    cases in (1) the set which yield an identical output and (2) the set which yield a different output.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    Qualitative testing can be used to assess the following:&#xD;
&lt;/p>&#xD;
&lt;ul style=&quot;MARGIN-TOP: 0cm&quot; type=&quot;disc&quot;>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        &lt;b style=&quot;mso-bidi-font-weight: normal&quot;>Correctness of a rules refactoring exercise&lt;/b>: as in regular software&#xD;
        lifecycle, rule set lifecycle includes refactoring where the rules are refined, the rule packages restructured and&#xD;
        the rule flow redesigned in order to accommodate the application needs for extensions or performances. After the&#xD;
        refactoring is complete, we should test that all possible input produces the same output through the champion and&#xD;
        challenger rule set. The refactoring exercise is successful if none of the test cases output differ between the&#xD;
        champion (original) and the challenger (refactored) rule sets.&#xD;
    &lt;/li>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        &lt;b style=&quot;mso-bidi-font-weight: normal&quot;>Stability of the system and extent of change:&lt;/b> after a business&#xD;
        requirement change has been implemented in a rule set, it is easy to come up with a set of test cases that should&#xD;
        show some impact from the change and another set that should not shown any impact from the change. By running the&#xD;
        latter set through delta-testing, we can decide on the stability of the system through the change. By running the&#xD;
        former, we can verify that the region of the solution space that we intended to update was properly targeted.&#xD;
    &lt;/li>&#xD;
&lt;/ul>&#xD;
&lt;h4>&#xD;
    &lt;a>Quantitative Delta Testing&lt;/a>&#xD;
&lt;/h4>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    By using quantitative delta-testing, i.e. by actually examining the difference in value between the output from the&#xD;
    champion and the challenger rule sets, we can achieve more specific testing and increase the usefulness and coverage of&#xD;
    our testing.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    Here, instead of using a strict equality criterion, we are testing that the difference between the value from the&#xD;
    champion output and the value from the challenger output is equal to a certain expectation such as a constant offset&#xD;
    value.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    For example, the base rates for mortgage products are often required to change. To test that the change is correctly&#xD;
    implemented, we simply need an equivalence function which compares the difference between the base rate of the champion&#xD;
    and the challenger and compares it to the base rate difference that should be introduced by the rules change.&#xD;
&lt;/p>&lt;br class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot; />&#xD;
&lt;br />&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    While requirement changes are frequent in business rules, they are often repetitive in nature and can be categorized in&#xD;
    a set of recurrent types of changes, such as change in base rate or, change in the set of underwriting conditions, etc…&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    So, after a few cycles of requirement changes, we end up with a set equivalence functions implemented as a &quot;comparison&#xD;
    tool-box&quot; that we can reuse to test most of the future requirement changes.&#xD;
&lt;/p>&#xD;
&lt;h3>&#xD;
    &lt;a>Intensive Test Cases Definition&lt;/a>&#xD;
&lt;/h3>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    We have shown in the previous sections that by using delta-testing, we can perform some quality assurance without&#xD;
    having to deal with expected values. By using intensive test case definitions from domains, we now show how we can&#xD;
    generate large amount of test cases from concise descriptions.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    In most cases, the variable values in a business application either belong to a finite discrete domain (e.g. &lt;span&#xD;
    class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>PropertyType&lt;/font>&lt;/span>&lt;/span> in &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>[SingleFamily, Condominium, Townhouse,&#xD;
    …]&lt;/font>&lt;/span>&lt;/span>) or can be discretized in buckets from which a representative value can be selected (e.g. &lt;span&#xD;
    class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>CreditScore&lt;/font>&lt;/span>&lt;/span> in &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>[499, 500, 550, …]&lt;/font>&lt;/span>&lt;/span>&#xD;
    &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 12pt; FONT-FAMILY: 'Times New Roman'; mso-bidi-font-family: 'Courier New'&quot;>or&lt;/span>&lt;/span>&lt;span&#xD;
    class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>LTV&lt;/font>&lt;/span>&lt;/span> &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 12pt; FONT-FAMILY: 'Times New Roman'; mso-bidi-font-family: 'Courier New'&quot;>in&lt;/span>&lt;/span>&lt;span&#xD;
    class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>[60.0, 65.0,&#xD;
    70.0, …]&lt;/font>&lt;/span>&lt;/span>).&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    This aspect is key in setting up intensive (versus extensive) descriptions of large sets of test case input data. An&#xD;
    extensive description of the test cases input values consists in listing all the values for all the variables involved&#xD;
    in the input data set. For example, test cases can be defined by the following tuples:&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;Domain&quot; style=&quot;MARGIN: 6pt 0cm 0pt 36pt&quot;>&#xD;
    &lt;font face=&quot;Courier New&quot; size=&quot;3&quot;>TC1 = {{CreditScore, 700}, {PropertyType, Condominium}, …}&lt;/font>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;Domain&quot; style=&quot;MARGIN: 6pt 0cm 0pt 36pt&quot;>&#xD;
    &lt;font face=&quot;Courier New&quot; size=&quot;3&quot;>TC2 = {{CreditScore, 700}, {PropertyType, SingleFamily}, …}&lt;/font>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;Domain&quot; style=&quot;MARGIN: 6pt 0cm 0pt 36pt&quot;>&#xD;
    &lt;font face=&quot;Courier New&quot; size=&quot;3&quot;>TC3 = {{CreditScore, 750}, {PropertyType, Condominium}, …}&lt;/font>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;Domain&quot; style=&quot;MARGIN: 6pt 0cm 0pt 36pt&quot;>&#xD;
    &lt;font face=&quot;Courier New&quot; size=&quot;3&quot;>TC4 = {{CreditScore, 750}, {PropertyType, SingleFamily}, …}&lt;/font>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    An intensive description consists in listing all the desired values for each variable (sub-domains), and adjoining a&#xD;
    strategy that traverses the cross-product of these sub-domains to generate the extensive version of the test cases. For&#xD;
    example, the above 4 test cases can be represented intensively by the following description.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;Domain&quot; style=&quot;MARGIN: 6pt 0cm 0pt 36pt&quot;>&#xD;
    &lt;font face=&quot;Courier New&quot; size=&quot;3&quot;>TC&lt;sub>1...4&lt;/sub> = {{CreditScore,{700, 750}},&lt;/font>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;Domain&quot; style=&quot;MARGIN: 6pt 0cm 0pt 72pt; TEXT-INDENT: 36pt&quot;>&#xD;
    &lt;font size=&quot;3&quot;>&lt;font face=&quot;Courier New&quot;>{PropertyType,{Condominium, SingleFamily}}, …}&lt;/font>&lt;/font>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    This definition is concise and the number of test cases it represents grows exponentially by adding values in the&#xD;
    sub-domains. For example, adding 2 values to the &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>PropertyType&lt;/font>&lt;/span>&lt;/span>&#xD;
    sub-domain and 3 values to the &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>CreditScore&lt;/font>&lt;/span>&lt;/span> leads to 20 different test cases instead of 4.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    Let us suppose that a guideline change for the &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>Condominium&lt;/font>&lt;/span>&lt;/span> property&#xD;
    type has been implemented and we need to test the updated rule set. We know that for all property type other than &lt;span&#xD;
    class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>Condominium&lt;/font>&lt;/span>&lt;/span>, no change in the output should occur, and that for the &lt;span&#xD;
    class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>Condominium&lt;/font>&lt;/span>&lt;/span> type, two specific conditions should be added to the output. We can&#xD;
    thus devise 2 sets of test cases:&#xD;
&lt;/p>&#xD;
&lt;ul style=&quot;MARGIN-TOP: 0cm&quot; type=&quot;disc&quot;>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        The first set has a sub-domain reduced to &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>[Condominium]&lt;/font>&lt;/span>&lt;/span> for&#xD;
        &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>PropertyType&lt;/font>&lt;/span>&lt;/span>, and we'll make sure that, for test cases in this set, there&#xD;
        are 2 more conditions in the challenger output than in the champion output.&#xD;
    &lt;/li>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        The second set has a sub-domain for &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>PropertyType&lt;/font>&lt;/span>&lt;/span>&#xD;
        which includes all the possible property types except &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>Condominium&lt;/font>&lt;/span>&lt;/span>, and&#xD;
        we'll make sure that, for test cases in this set, the champion and challenger outputs are identical.&#xD;
    &lt;/li>&#xD;
&lt;/ul>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    Using intensive test case definition is completing the delta-testing approach by allowing the creation of large number&#xD;
    of test case input data without having to spell out each individual test case. The intensive definition prescribes&#xD;
    which possible values can be used in the test case data. The actual construction of the test cases is left to an&#xD;
    automated enumeration mechanism.&#xD;
&lt;/p>&#xD;
&lt;h3>&#xD;
    &lt;a>Extensive Test Case Definition&lt;/a>&#xD;
&lt;/h3>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    One of the best sources of test cases for systems running in production is obviously the application database which&#xD;
    logs the system client transactions. The most significant business data combinations can be obtained in large numbers&#xD;
    from this database.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    Delta testing makes it possible to profit from this large set of test cases, mainly through quantitative delta testing.&#xD;
    Tens of thousand real life test cases can be extracted from the database and executed each time a new rule set is to be&#xD;
    released to verify the extent of the rule change, as described in the Qualitative Delta Testing section.&#xD;
&lt;/p>&#xD;
&lt;h3>&#xD;
    &lt;a>Sample Test Harness Implementation&lt;/a>&#xD;
&lt;/h3>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    We propose here some highlights of a sample practical implementation of the delta testing harness. It allows the&#xD;
    selection of a champion and a challenger rule set file, and the execution of a batch of test suites defined in a batch&#xD;
    file.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    For each test suite, two Excel (CSV) files are generated: one which records the test cases input data which produce an&#xD;
    equivalent output through the comparison function and the other which records the test case input data which produce a&#xD;
    non-equivalent output.&#xD;
&lt;/p>&#xD;
&lt;h4>&#xD;
    &lt;a>Batch File Definition&lt;/a>&#xD;
&lt;/h4>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The batch file is composed of a set of test suite definitions. Each line of the batch file can either be blank, or a&#xD;
    comment line (line which first non-blank character is #) or a test suite definition of one of the two forms below:&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    &lt;span class=&quot;DomainChar&quot;>&lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 10.0pt&quot;>&lt;font face=&quot;Courier New&quot;>INT &amp;lt;output&amp;gt; &amp;lt;domains&amp;gt;&#xD;
    &amp;lt;order&amp;gt; &amp;lt;count&amp;gt; [&amp;lt;comparator&amp;gt; [&amp;lt;arg&lt;sub>1&lt;/sub>&amp;gt; …&#xD;
    &amp;lt;arg&lt;sub>n&lt;/sub>&amp;gt;]]&lt;/font>&lt;/span>&lt;/b>&lt;/span>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    &lt;span class=&quot;DomainChar&quot;>&lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 10.0pt&quot;>&lt;font face=&quot;Courier New&quot;>EXT &amp;lt;output&amp;gt; &amp;lt;testcases&amp;gt;&#xD;
    &amp;lt;count&amp;gt; [&amp;lt;comparator&amp;gt; [&amp;lt;arg&lt;sub>1&lt;/sub>&amp;gt; … &amp;lt;arg&lt;sub>n&lt;/sub>&amp;gt;]]&lt;/font>&lt;/span>&lt;/b>&lt;/span>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The &lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
    style=&quot;FONT-FAMILY: 'Courier New'; mso-bidi-font-size: 10.0pt&quot;>INT&lt;/span>&lt;/b> prefix is for test suites using intensive&#xD;
    test case definition, while &lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
    style=&quot;FONT-FAMILY: 'Courier New'; mso-bidi-font-size: 10.0pt&quot;>EXT&lt;/span>&lt;/b> is for test suites using extensive test&#xD;
    case definition.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The test suite parameters definitions are:&#xD;
&lt;/p>&#xD;
&lt;ul style=&quot;MARGIN-TOP: 0cm&quot; type=&quot;disc&quot;>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        &lt;span class=&quot;DomainChar&quot;>&lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>&amp;lt;output&amp;gt;&lt;/font>&lt;/span>&lt;/b>&lt;/span>: the prefix used to generate the name of the Excel&#xD;
        files. Two files named &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>&amp;lt;output&amp;gt;_EQUIVALENT.csv&lt;/font>&lt;/span>&lt;/span> and &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>&amp;lt;output&amp;gt;_DIFFERENT.csv&lt;/font>&lt;/span>&lt;/span> are generated as the result of running the&#xD;
        test suite.&#xD;
    &lt;/li>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        &lt;span class=&quot;DomainChar&quot;>&lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>&amp;lt;domains&amp;gt;&lt;/font>&lt;/span>&lt;/b>&lt;/span>: the name of the file which contains the definition of&#xD;
        the sub-domains of values for each necessary variable. Each line of a domain file should either be blank, or a&#xD;
        comment line, or a sub-domain definition of the form:&#xD;
    &lt;/li>&#xD;
&lt;/ul>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm 3pt 36pt&quot;>&#xD;
    &lt;font face=&quot;Courier New&quot;>&lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&amp;lt;variable&amp;gt; = &amp;lt;value&lt;/span>&lt;/span>&lt;span&#xD;
    class=&quot;DomainChar&quot;>&lt;sub>&lt;span style=&quot;FONT-SIZE: 11pt&quot;>1&lt;/span>&lt;/sub>&lt;/span>&lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&amp;gt; … &amp;lt;value&lt;/span>&lt;/span>&lt;span class=&quot;DomainChar&quot;>&lt;sub>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt&quot;>n&lt;/span>&lt;/sub>&lt;/span>&lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&amp;gt;&lt;/span>&lt;/span>&lt;/font>&#xD;
&lt;/p>&#xD;
&lt;ul style=&quot;MARGIN-TOP: 0cm&quot; type=&quot;disc&quot;>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        &lt;span class=&quot;DomainChar&quot;>&lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span style=&quot;FONT-SIZE: 11pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>&amp;lt;testcases&amp;gt;&lt;/font>&lt;/span>&lt;/b>&lt;/span>&lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 12pt; FONT-FAMILY: 'Times New Roman'; mso-bidi-font-family: 'Courier New'&quot;>: the name of the CSV&#xD;
        file which contains one test case definition (set of attribute values) per row.&lt;/span>&lt;/span>&#xD;
    &lt;/li>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        &lt;span class=&quot;DomainChar&quot;>&lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>&amp;lt;order&amp;gt;&lt;/font>&lt;/span>&lt;/b>&lt;/span>: the order in which the sub-domains should be traversed&#xD;
        to generate the test cases. The value is one of &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>DFS&lt;/font>&lt;/span>&lt;/span> or &lt;span&#xD;
        class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>RANDOM&lt;/font>&lt;/span>&lt;/span>. The &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>DFS&lt;/font>&lt;/span>&lt;/span> order will&#xD;
        traverse the set of domains in depth first order, thus guarantying that all tuples are visited once and only once.&#xD;
        &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>RANDOM&lt;/font>&lt;/span>&lt;/span> order will repeatedly choose a value at random in each sub-domain.&#xD;
    &lt;/li>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        &lt;span class=&quot;DomainChar&quot;>&lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>&amp;lt;count&amp;gt;&lt;/font>&lt;/span>&lt;/b>&lt;/span>: the number of test cases to submit for evaluation out of&#xD;
        the ones that can be generated from the sub-domains. In the case of &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>DFS&lt;/font>&lt;/span>&lt;/span> order, if the&#xD;
        size of the cross-product of the sub-domains is smaller than the requested count, the execution stops after all&#xD;
        test cases from the cross-product have been submitted for evaluation.&#xD;
    &lt;/li>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        &lt;span class=&quot;DomainChar&quot;>&lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>&amp;lt;comparator&amp;gt;&lt;/font>&lt;/span>&lt;/b>&lt;/span> is an optional parameter which is the name of the&#xD;
        comparator class to use to evaluate the equivalence of outputs. The class should implement a common interface&#xD;
        defining the equivalence function (see class &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
        face=&quot;Courier New&quot;>ResponseComparator&lt;/font>&lt;/span>&lt;/span> in the section).&#xD;
    &lt;/li>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l3 level1 lfo2; tab-stops: list 36.0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>&lt;span class=&quot;DomainChar&quot;>&lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&amp;lt;arg&lt;/span>&lt;/b>&lt;/span>&lt;span class=&quot;DomainChar&quot;>&lt;b&#xD;
        style=&quot;mso-bidi-font-weight: normal&quot;>&lt;sub>&lt;span style=&quot;FONT-SIZE: 11pt&quot;>i&lt;/span>&lt;/sub>&lt;/b>&lt;/span>&lt;span&#xD;
        class=&quot;DomainChar&quot;>&lt;b style=&quot;mso-bidi-font-weight: normal&quot;>&lt;span&#xD;
        style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&amp;gt;&lt;/span>&lt;/b>&lt;/span>&lt;/font> are optional additional arguments&#xD;
        to pass to the constructor of the comparator class. If no comparator parameter is specified, a default &quot;strict&#xD;
        equality&quot; comparator class is used.&#xD;
    &lt;/li>&#xD;
&lt;/ul>&#xD;
&lt;h4>&#xD;
    &lt;a>Batch File Example&lt;/a>&#xD;
&lt;/h4>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm 6pt&quot;>&#xD;
    Below is an example of the content of a batch file with 4 test suites, testing 2 main business requirement changes:&#xD;
&lt;/p>&#xD;
&lt;div&#xD;
style=&quot;BORDER-RIGHT: windowtext 1pt solid; PADDING-RIGHT: 4pt; BORDER-TOP: windowtext 1pt solid; PADDING-LEFT: 4pt; PADDING-BOTTOM: 1pt; BORDER-LEFT: windowtext 1pt solid; PADDING-TOP: 1pt; BORDER-BOTTOM: windowtext 1pt solid; mso-border-alt: solid windowtext .5pt; mso-element: para-border-div&quot;>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;># Test suite for release 2.5.1&lt;/font>&#xD;
    &lt;/p>&lt;br class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot; />&#xD;
    &lt;br />&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;># Testing requirement &quot;decrease base rate by 15bps&quot;&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>#&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>INT base rel251.domains RANDOM 1000 BaseRateComparator -0.15&lt;/font>&#xD;
    &lt;/p>&lt;br class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot; />&#xD;
    &lt;br />&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;># Testing requirement &quot;decrease minimum rate for FIXED products by 25bps&quot;&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>#&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>INT min15x15 rel251.domains RANDOM 500 MinimumRateComparator FIXED15x15 -0.25&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>INT min30x30 rel251.domains RANDOM 500 MinimumRateComparator FIXED30x30 -0.25&lt;/font>&#xD;
    &lt;/p>&lt;br class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot; />&#xD;
    &lt;br />&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;># Running regression tests&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>#&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>EXT regression regression.csv 5000&lt;/font>&#xD;
    &lt;/p>&#xD;
&lt;/div>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    This example shows the use of 2 comparator classes: &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>BaseRateComparator&lt;/font>&lt;/span>&lt;/span>&#xD;
    and &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>MinimumRateComparator&lt;/font>&lt;/span>&lt;/span>. These comparators (equivalence functions) are dedicated&#xD;
    to test respectively a change (increase or decrease) of base rates and a change (increase or decrease) of minimum rates&#xD;
    for a given mortgage product. Once implemented, these comparators can be reused for all such routine changes.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    While the first 3 test suites are using an intensive test case definition through a domain file, the last test suite&#xD;
    uses the &lt;span style=&quot;FONT-FAMILY: 'Courier New'; mso-bidi-font-size: 10.0pt&quot;>regression.csv&lt;/span> file as the set of&#xD;
    extensive test case definition. This test suite does not specify any custom comparator and will thus use the default&#xD;
    strict equality.&#xD;
&lt;/p>&#xD;
&lt;h4>&#xD;
    &lt;a>Domains File Example&lt;/a>&#xD;
&lt;/h4>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm 6pt&quot;>&#xD;
    Below is an excerpt of a domains file:&#xD;
&lt;/p>&#xD;
&lt;div&#xD;
style=&quot;BORDER-RIGHT: windowtext 1pt solid; PADDING-RIGHT: 4pt; BORDER-TOP: windowtext 1pt solid; PADDING-LEFT: 4pt; PADDING-BOTTOM: 1pt; BORDER-LEFT: windowtext 1pt solid; PADDING-TOP: 1pt; BORDER-BOTTOM: windowtext 1pt solid; mso-border-alt: solid windowtext .5pt; mso-element: para-border-div&quot;>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>State = CALIFORNIA MARYLAND VIRGINIA DELAWARE HAWAII MICHIGAN FLORIDA LOUISIANA&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>PropertyValue = 60000 100000 110000 120000 150000 160000 200000 300000 400000&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>Ltv = 65 70 75 80 85 90 95 100&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>CreditScore = 500 580 600 639 640 700 720 750 760&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>DocumentationType = FullDocumentation NoDocumentation Reduced&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>PurchaseType = Purchase Refinance Other&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>PropertyType = SingleFamily Condominium Townhouse Modular PUD&lt;/font>&#xD;
    &lt;/p>&#xD;
    &lt;p class=&quot;code&quot; style=&quot;MARGIN: 0cm 0cm 0pt&quot;>&#xD;
        &lt;font face=&quot;Courier New&quot;>PropertyUsageType = Investor Primary Residence SecondHome&lt;/font>&#xD;
    &lt;/p>&#xD;
&lt;/div>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    Note that when using the &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>RANDOM&lt;/font>&lt;/span>&lt;/span> traversal order for domain, the same value can be repeated multiple&#xD;
    times for a given variable, so that the probability distribution is different for different values.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    For example, if we want to reflect that most of the time, the &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>PropertyType&lt;/font>&lt;/span>&lt;/span> value is&#xD;
    &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>SFR&lt;/font>&lt;/span>&lt;/span>, we can specify the following sub-domain:&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;Domain&quot; style=&quot;MARGIN: 6pt 0cm 0pt&quot;>&#xD;
    &lt;font face=&quot;Courier New&quot; size=&quot;3&quot;>PropertyType = SingleFamily Condominium Modular PUD PUD PUD&lt;/font>&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    &lt;a>This example gives&lt;/a> &lt;span style=&quot;mso-bookmark: _Ref106618144&quot;>&lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>PUD&lt;/font>&lt;/span>&lt;/span> a 50% chance of&#xD;
    being selected in the test cases generation process versus a 16.7% chance for the other property types..&lt;/span>&#xD;
&lt;/p>&#xD;
&lt;h3>&#xD;
    &lt;span style=&quot;mso-bookmark: _Ref106618144&quot;>&lt;a>Test Harness Design&lt;/a>&lt;/span>&#xD;
&lt;/h3>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The class structure of the test harness that supports delta-testing is illustrated on &lt;span&#xD;
    style=&quot;mso-field-code: ' REF _Ref106678587 h '&quot;>Figure &lt;span style=&quot;mso-no-proof: yes&quot;>3&lt;/span>&lt;/span>. &lt;span&#xD;
    class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>TestHarness&lt;/font>&lt;/span>&lt;/span> is the central class from which the rule sets are loaded and the&#xD;
    batch test file is run. It uses 2 instances of &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>RuleHarness&lt;/font>&lt;/span>&lt;/span>, one for&#xD;
    the champion, one for the challenger, which are representing the 2 engine instances.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>TestCaseIterator&lt;/font>&lt;/span>&lt;/span> class is an abstract class with 2 concrete subclasses: &lt;span&#xD;
    style=&quot;FONT-FAMILY: 'Courier New'; mso-bidi-font-size: 10.0pt&quot;>IntensiveTestCaseIterator&lt;/span> is responsible for&#xD;
    implementing an iterator which, given a domains file, iterates over the tuples of values which can be produced from the&#xD;
    sub-domains. ExtensiveTestCaseIterator is responsible for implementing an iterator through the tuples of values defined&#xD;
    by each row of a given CSV file. A tuple is represented by an instance of the &lt;span class=&quot;DomainChar&quot;>&lt;span&#xD;
    style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font face=&quot;Courier New&quot;>RequestHolder&lt;/font>&lt;/span>&lt;/span> class,&#xD;
    which provides a mapping function to the actual request object from the application BOM.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    The &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>ResponseComparator&lt;/font>&lt;/span>&lt;/span> class is an abstract class which declares the equivalence&#xD;
    function &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>areEquivalent&lt;/font>&lt;/span>&lt;/span> between 2 instances of the response object from the application&#xD;
    BOM. The &lt;span class=&quot;DomainChar&quot;>&lt;span style=&quot;FONT-SIZE: 11pt; mso-bidi-font-size: 12.0pt&quot;>&lt;font&#xD;
    face=&quot;Courier New&quot;>ExactComparator&lt;/font>&lt;/span>&lt;/span> subclass performs the strict equality comparison between 2&#xD;
    response objects. Other specific comparator classes are implemented for quantitative testing.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    Note that our sample implementation of the test harness is making a heavy use of Java reflection to traverse and&#xD;
    compare the content of the responses from the rule engines.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm; TEXT-ALIGN: center&quot; align=&quot;center&quot;>&#xD;
    &amp;nbsp;&lt;img height=&quot;544&quot; alt=&quot;&quot; src=&quot;resources/harness_class_diagram.gif&quot; width=&quot;600&quot; />&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoCaption&quot; style=&quot;MARGIN: 6pt 0cm 6pt 11.35pt; TEXT-ALIGN: center&quot; align=&quot;center&quot;>&#xD;
    &lt;a>&lt;strong>Figure&lt;/strong>&lt;/a> &lt;strong>&lt;span style=&quot;mso-bookmark: _Ref106678587&quot;>&lt;span&#xD;
    style=&quot;mso-no-proof: yes&quot;>3&lt;/span>&lt;/span>: Test Harness Class Diagram&lt;/strong>&#xD;
&lt;/p>&#xD;
&lt;h3>&#xD;
    &lt;a>Conclusion&lt;/a>&#xD;
&lt;/h3>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    While business rules based applications usually receive proper testing on their initial release, we often fail to plan&#xD;
    for (and even sometime to realize) that:&#xD;
&lt;/p>&#xD;
&lt;ul style=&quot;MARGIN-TOP: 0cm&quot; type=&quot;disc&quot;>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l0 level1 lfo3; tab-stops: list 36.0pt&quot;>&#xD;
        The initial business requirements will change because it is the essence the business rules based systems, and test&#xD;
        cases will have to be revised accordingly.&#xD;
    &lt;/li>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l0 level1 lfo3; tab-stops: list 36.0pt&quot;>&#xD;
        The rapid pace of changes is essential to the organization and while it can usually be sustained on the&#xD;
        implementation side, it should not be slowed-down by the QA process.&#xD;
    &lt;/li>&#xD;
&lt;/ul>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    Add to this the scarcity of SMEs able to maintain the test case base, and we obviously need a testing paradigm that is&#xD;
    not solely based on the comparison of &lt;i style=&quot;mso-bidi-font-style: normal&quot;>actual&lt;/i> against &lt;i&#xD;
    style=&quot;mso-bidi-font-style: normal&quot;>expected&lt;/i> results.&#xD;
&lt;/p>&#xD;
&lt;p class=&quot;MsoNormal&quot; style=&quot;MARGIN: 3pt 0cm&quot;>&#xD;
    We proposed a framework that is based instead on the comparison of &lt;i style=&quot;mso-bidi-font-style: normal&quot;>champion&lt;/i>&#xD;
    against &lt;i style=&quot;mso-bidi-font-style: normal&quot;>challenger&lt;/i> results. Among the expected benefits are:&#xD;
&lt;/p>&#xD;
&lt;ul style=&quot;MARGIN-TOP: 0cm&quot; type=&quot;disc&quot;>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l2 level1 lfo4; tab-stops: list 36.0pt&quot;>&#xD;
        To free the testing team from the burden of maintaining expected results, thus accelerating the testing process and&#xD;
        reducing the bottleneck for rapid change deployment.&#xD;
    &lt;/li>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l2 level1 lfo4; tab-stops: list 36.0pt&quot;>&#xD;
        To makes test writing more technical, thus more manageable by a business-agnostic QA department.&#xD;
    &lt;/li>&#xD;
    &lt;li class=&quot;MsoNormal&quot; style=&quot;MARGIN: 6pt 0cm 0pt; mso-list: l2 level1 lfo4; tab-stops: list 36.0pt&quot;>&#xD;
        To support the production and execution of high volume of test cases through intensive test case definitions, thus&#xD;
        improving the coverage of the QA process.&#xD;
    &lt;/li>&#xD;
&lt;/ul>&#xD;
&lt;div style=&quot;mso-element: footnote-list&quot;>&#xD;
    &lt;br clear=&quot;all&quot; />&#xD;
    &lt;hr align=&quot;left&quot; width=&quot;33%&quot; size=&quot;1&quot; />&#xD;
    &lt;div id=&quot;ftn1&quot; style=&quot;mso-element: footnote&quot;>&#xD;
        &lt;p class=&quot;MsoFootnoteText&quot; style=&quot;MARGIN: 6pt 0cm 0pt&quot;>&#xD;
            &lt;font face=&quot;Times New Roman&quot;>&lt;u>&lt;font color=&quot;#800080&quot;>(1)&lt;/font>&lt;/u> We use the term test suite for a&#xD;
            collection of test cases dedicated to the test of a specific system feature.&lt;/font>&#xD;
        &lt;/p>&#xD;
    &lt;/div>&#xD;
&lt;/div></mainDescription>
</org.eclipse.epf.uma:ContentDescription>