;This is the September 15, 1991 version of ;the on-line documentation for Richard Waters' test case coverage checker. The COVER system: Determining the Coverage of a Test Suite Richard C. Waters MIT AI Laboratory 545 Technology Sq. Cambridge MA 02139 Dick@AI.MIT.EDU Mitsubishi Electric Research Laboratories 201 Broadway Cambridge MA 02139 Dick@MERL.COM The value of a suite of test cases depends critically on its coverage. Ideally a suite should test every facet of the specification for a program and every facet of the algorithms used to implement the specification. Unfortunately, there is no practical way to be sure that complete coverage has been achieved. However, something should be done to assess the coverage of a test suite, because a test suite with poor coverage has little value. A traditional approximate method of assessing the coverage of a test suite is to check that every condition tested by the program is exercised. For every predicate in the program, there should be at least one test case that causes the predicate to be true and one that causes it to be false. Consider the function MY* below which uses a convoluted algorithm to compute the product of two numbers. (DEFUN MY* (X Y) (LET ((SIGN 1)) (WHEN (MINUSP X) (SETQ SIGN (- SIGN)) (SETQ X (- X))) (WHEN (MINUSP Y) (SETQ SIGN (- SIGN)) (SETQ Y (- X))) (* SIGN X Y))) The function my* contains two predicates, (MINUSP X) and (MINUSP Y), which lead to four conditions: X is negative, X is not negative, Y is negative, and Y is not negative. To be at all thorough, a test suite must contain tests exercising all four of these conditions. For instance, any test suite that fails to exercise the condition where Y is negative will fail to detect the bug in the next to last line of the function. (As an example of the fact that covering all the conditions in a program does not guarantee that every facet of either the algorithm or the specification will be covered, consider the fact that the two test cases (MY* 2.1 3) and (MY* -1/2 -1/2) cover all four conditions. However, they do not detect the bug on the next to last line and they do not detect the fact that MY* fails to work on complex numbers.) The COVER system determines which conditions tested by a program are exercised by a given test suite. This is no substitute for thinking hard about the coverage of the test suite. However, it provides a useful starting point and can indicate some areas where additional test cases should be devised. USER'S MANUAL FOR COVER The functions, macros, and variables that make up the COVER system are in a package called "COVER". The six exported symbols are documented below. COVER:ANNOTATE t-or-nil Evaluating (COVER:ANNOTATE T) triggers the processing of function and macro definitions by the COVER system. Each subsequent instance of DEFUN or DEFMACRO is altered by adding annotation that maintains information about the various conditions tested in the body. Evaluating (COVER:ANNOTATE NIL) stops the special processing of function and macro definitions. Subsequent definitions are not annotated. However, if a function or macro that is currently annotated is redefined, the new definition is annotated as well. The macro COVER:ANNOTATE should only be used as a top-level form. When annotation is triggered, a warning message is printed, and T is returned. Otherwise, NIL is returned. (COVER:ANNOTATE T) => t ; after printing: ;;; Warning: Coverage annotation applied. COVER:FORGET-ALL This function, which always returns t, has the effect of removing all coverage annotation from every function and macro. It is appropriate to do this before completely recompiling the system being tested or before switching to a different system to be tested. COVER:RESET Each condition tested by an annotated function and macro is associated with a flag that trips when the condition is exercised. The function COVER:RESET resets all these flags, and returns T. It is appropriate to do this before rerunning a test suite to reevaluate its coverage. COVER:REPORT &KEY FN OUT ALL This function displays the information maintained by COVER, returning no values. FN must be the name of an annotated function or macro. If FN is specified, a report is printed showing information about that function or macro only. Otherwise, reports are printed about every annotated function and macro. OUT, which defaults to *STANDARD-OUTPUT*, must either be an output stream or the name of a file. It specifies where the reports should be printed. If ALL, which defaults to NIL, is non-null then the reports printed contain information about every condition. Otherwise, the reports are abbreviated to highlight key conditions that have not been exercised. COVER:*LINE-LIMIT* default value 75 The output produced by COVER:REPORT is truncated to ensure that it is no wider than COVER:*LINE-LIMIT*. AN EXAMPLE Suppose that the function MY* has been annotated and that no other functions or macros have been annotated. The following example illustrates the operation of COVER and the reports printed by COVER:REPORT. (setq cover:*line-limit* 43) => 43 (cover:reset) => T (cover:report) => ; after printing: ;- :REACH (DEFUN MY* (X Y)) <1> (my* 2 2) => 4 (cover:report) => ; after printing: ;+ :REACH (DEFUN MY* (X Y)) <1> ; + :REACH (WHEN (MINUSP X) (SETQ S <2> ; - :NON-NULL (MINUSP X) <4> ; + :REACH (WHEN (MINUSP Y) (SETQ S <6> ; - :NON-NULL (MINUSP Y) <8> (my* -2 2) => -4 (cover:report) => ; after printing: ;+ :REACH (DEFUN MY* (X Y)) <1> ; + :REACH (WHEN (MINUSP Y) (SETQ S <6> ; - :NON-NULL (MINUSP Y) <8> (cover:report :all t) => ; after printing: ;+ :REACH (DEFUN MY* (X Y)) <1> ; + :REACH (WHEN (MINUSP X) (SETQ S <2> ; + :NON-NULL (MINUSP X) <4> ; + :NULL (MINUSP X) <5> ; + :REACH (WHEN (MINUSP Y) (SETQ S <6> ; - :NON-NULL (MINUSP Y) <8> ; + :NULL (MINUSP Y) <9> Each line in a report contains three pieces of information about a point in a definition: +/- specifying that the point either has (+) or has not (-) been exercised, a message indicating the physical and logical placement of the point in the definition, and in angle brackets < >, an integer that is a unique identifier for the point. Indentation is used to indicate that some points are subordinate to others in the sense that the subordinate points cannot be exercised without also exercising their superiors. The order of the lines of the report is the same as the order of the points in the definition. Each message contains a label (e.g., :REACH, :NULL) and a piece of code. There is a point labeled :REACH corresponding to each definition as a whole and each conditional form within each definition. Subordinate points corresponding to the conditions a conditional form tests are grouped under the point corresponding to the form. As discussed in detail in the next subsection, the messages for the subordinate points describe the situations in which the conditions are exercised. Lines that would otherwise be too long to fit on one line have their messages truncated (e.g., points <2> and <6>). The first three reports are abbreviated based on two principles. First, if a point P and all of its subordinates have been exercised, then P and all of its subordinates are omitted from the report. This is done to focus the user's attention on the points that have not been exercised. Second, if a point P has not been exercised, then all of the points subordinate to it are omitted from the report. This reflects the fact that it is not possible for any of these subordinate points to have been exercised and one cannot devise a test case that exercises any of the subordinate points without first figuring out how to exercise P. An additional complicating factor is that COVER operates in an incremental fashion and does not, in general, have full information about the subordinates of points that have not been exercised. As a result, it is not always possible to present a complete report. However, one can have total confidence that if the report says that every point has been exercised, this statement is based on complete information. The first report in shows that none of the points within MY* has been exercised. The second report displays most of the points in MY*, to set the context for the two points that have not been exercised. The third report omits <2> and its subordinates, since they have all been exercised. The fourth report shows a complete report corresponding to the third abbreviated report. COVER:FORGET &rest ids This function gives the user greater control over the reports produced by COVER:REPORT. Each ID must be an integer identifying a point. All information about the specified points (and their subordinates) is forgotten. From the point of view of COVER:REPORT, the effect is as if the points never existed. (A forgotten point can be retrieved by reevaluating or recompiling the function or macro definition containing it.) The example below, which follows on after the end of the example above, shows the action of COVER:FORGET. (cover:forget 6) => T (cover:report :all t) => ; after printing: ;+ :REACH (DEFUN MY* (X Y)) <1> ; + :REACH (WHEN (MINUSP X) (SETQ S <2> ; + :NON-NULL (MINUSP X) <4> ; + :NULL (MINUSP X) <5> (cover:report) => after printing ;All points exercised. The abbreviated report above does not describe any points, because every point in MY* that has not been forgotten has been exercised. It is appropriate to forget a point if there is some reason that no test case can possibly exercise the point. However, it is much better to write your code so that every condition can be tested. (Point numbers are assigned based on the order in which points are entered into COVER's database. In general, whenever a definition is reevaluated or recompiled, the numbers of the points within it change.) THE WAY CONDITIONALS ARE ANNOTATED <If the follwing is written in a file> (in-package "USER") (require "COVER" ...) (defmacro maybe+ (x y) `(if (numberp ,x) (+ ,x ,y))) (cover:annotate t) (defmacro maybe- (x y) `(if (numberp ,x) (- ,x ,y))) (defun g (x y) (cond ((and (null x) y) y) (y (case y (1 (maybe- x y)) (2 (maybe+ x y)))))) (cover:annotate nil) (defun h (x y) ...) (cover:reset) (run-tests) (cover:report :out "report" :all t) <loading the file produces the following output in the file "report"> ;+ :REACH (DEFMACRO MAYBE- (X Y)) <1> ;+ :REACH (DEFUN G (X Y)) <2> ; + :REACH (COND ((AND # Y) Y) (Y ( <3> ; + :REACH (AND (NULL X) Y) <9> ; + :FIRST-NULL (NULL X) <11> ; + :EVAL-ALL Y <12> ; + :FIRST-NON-NULL (AND (NULL X) <5> ; + :FIRST-NON-NULL Y <7> ; + :REACH (CASE Y (1 (MAYBE- X Y <13> ; + :SELECT 1 <15> ; + :REACH (IF (NUMBERP X) (- X <18> ; + :NON-NULL (NUMBERP X) <20> ; + :NULL (NUMBERP X) <21> ; + :SELECT 2 <16> ; - :SELECT-NONE <17> ; + :ALL-NULL <8> The above shows a file that makes use of COVER and the kind of report that might be produced by loading the file. Because, MAYBE- and G are the only definitions that have been annotated, these are the only definitions that are reported on. The order of the reports is the same as the order in which the definitions were compiled. The report on G indicates that the tests performed by RUN-TESTS exercise most of the conditions tested by G. However, they do not exercise the situation in which the CASE statement is reached, but neither of its clauses is selected. There are no points within MAYBE-, because the code for MAYBE- does not contain any conditional forms. It is interesting to consider the precise points that COVER includes for G. When COVER processes a definition, a cluster of points is generated corresponding to each conditional form (i.e., IF, WHEN, UNTIL, COND, CASE, TYPECASE, AND, and OR) that is literally present in the program. In addition, points are generated corresponding to conditional forms that are produced by macros that are annotated (e.g., the IF produced by the MAYBE- in the first CASE clause in G). However, annotation is not applied to conditionals that come from other sources (e.g., from macros that are defined outside of the system being tested). These conditionals are omitted, because there is no reasonable way for the user to know how they relate to the code, and therefore there is no reasonable way for the user to devise a test case that will exercise them. The messages associated with a point's subordinates describe the situations under which the subordinates are exercised. The pattern of messages associated with CASE and TYPECASE is illustrated by the portion (reproduced below) of the report above that describes the CASE in G. ; + :REACH (CASE Y (1 (MAYBE- X Y <13> ; + :SELECT 1 <15> ; + :SELECT 2 <16> ; - :SELECT-NONE <17> There are two subpoints corresponding to the two clauses of the CASE. In addition, since the last clause does not begin with T or OTHERWISE, there is an additional point corresponding to the situation where none of the clauses of the CASE are executed. The pattern of messages associated with a COND is illustrated by the portion of the report that describes the COND in G. ; + :REACH (COND ((AND # Y) Y) (Y ( <3> ; + :REACH (AND (NULL X) Y) <9> ; + :FIRST-NON-NULL (AND (NULL X) <5> ; + :FIRST-NON-NULL Y <7> ; + :ALL-NULL <8> There are subordinate points corresponding to the two clauses and the situation where neither clause is executed. There is also a point <9> corresponding to the AND that is the predicate of the first COND clause. This point is placed directly under <3>, because it is not subordinate to any of the individual COND clauses. The treatment of AND (and OR) is particularly interesting. Sometimes AND is used as a control construct on a par with COND. In that situation, it is clear that AND should be treated analogously to COND. However, at other times, AND is used to compute a value that is tested by another conditional form. In that situation, COVER could choose to treat AND as a simple function. However, it is nevertheless still reasonable to think of an AND as having conditional points that correspond to different reasons why the AND returns a true or false value. It is wise to include tests corresponding to each of these different reasons. The pattern of messages associated with an AND is illustrated by the portion of the report that describes the AND in G. (cover:report :all t) ; + :REACH (AND (NULL X) Y) <9> ; + :FIRST-NULL (NULL X) <11> ; + :EVAL-ALL Y <12> The final subpoint corresponds to the situation where all of the arguments of the AND have been evaluated. The AND then returns whatever the final argument returned. The example in this section illustrates a batch-oriented use of COVER. However, COVER is most effectively used in an interactive way. It is recommended that you first create as comprehensive a test suite as you can and capture it using a tool such as RT. The tests should then be run in conjunction with COVER and repeated reports from COVER generated as additional tests are created until complete coverage of conditions has been achieved. To robustly support this mode of operation, COVER has been carefully designed so that it will work with batch-compiled definitions, incrementally-compiled definitions, and interpreted definitions. OBTAINING COVER COVER is written in portable Common Lisp and has been tested in several different Common Lisp implementations. The source for cover can be obtained over the INTERNET by using FTP. Connection should be made to FTP.AI.MIT.EDU (INTERNET number 128.52.32.6). Login as ``anonymous'' and copy the files shown below. In the directory /pub/lptrs/ cover.lisp ; source code cover-test.lisp ; test suite cover-doc.txt ; brief documentation The code for COVER and the files above are copyright 1991 by the Massachusetts Institute of Technology, Cambridge MA. Permission to use, copy, modify, and distribute this software for any purpose and without fee is hereby granted, provided that this copyright and permission notice appear in all copies and supporting documentation, and that the names of MIT and/or the author are not used in advertising or publicity pertaining to distribution of the software without specific, written prior permission. MIT and the author make no representations about the suitability of this software for any purpose. It is provided ``as is'' without express or implied warranty. MIT and the author disclaim all warranties with regard to this software, including all implied warranties of merchantability and fitness. In no event shall MIT or the author be liable for any special, indirect or consequential damages or any damages whatsoever resulting from loss of use, data or profits, whether in an action of contract, negligence or other tortious action, arising out of or in connection with the use or performance of this software.