My Lisp Unit Test Framework

created Friday, 2005 June 17
updated Wednesday, 2005 December 28

1 What is this?

This is a description of the unit test framework I use for Lisp. It's called CyberTiggyr Test. Anyone is welcomed to use it, but this essay & source code are here mostly for my own use.

1.1 Other Unit Test Frameworks for Lisp

Mine is hardly the only one. There are also...

Chris Riesbeck's lisp-unit
Hierarchical Extensible Unit Testing Environment for Common Lisp (HEUTE)
probably many others.

2 Examples

Here are some example tests.¹

(load "test.lisp")
(import 'cybertiggyr-test:deftest)

(deftest test0000 ()
  "Null test.  Always succeeds.  Usefull when everything is
going wrong."
  'test0000)

(deftest test0010 ()
  "Ensure we can create a circle.  Use default values.  Do
not check that the circle is valid,just that we can create
one."
  (make-circle))

(deftest test-curvature ()
  (let ((circle (make-circle :radius 3)))
    (unless (= (/ (circumference circle) (radius circle) pi))
      ;; In a spherical field, pi has a different meaning...or
      ;; value, depending on whom you ask.
      (format t "~&We're not in Kansas any more."))
    (= (/ (circumference circle) (radius circle) pi))))

(deftest test0100 ()
  "Ensure the config file exists & we have permission to read it."
  (let* ((pathname (make-pathname :name "config" :type "lisp"))
         (is-good (with-open-file (strm pathname
                                       :if-does-not-exist nil)
                    (and strm (read strm)))))
    (unless is-good
      (format t "~&Could not open ~S for input." pathname)
      (format t "~&Does it exist?  Do we have permission to read it?"))
    is-good))

(cybertiggyr-test:run)                  ; run all the tests

3 License

CyberTiggyr Test is licensed according to the Gnu Lesser General Public License.

4 Obtaining

The source code is in a single file at http://cybertiggyr.com/gene/lut/test.lisp. It is also included in an appendix of the printed version of this essay.

5 Basic Usage

LOAD test.lisp into your Lisp environment, as part of the application you are developing, just like you'd load other source files.
test.lisp does not depend on anything other than Common Lisp, so you can load it early.²
Find a package in which to put some test programs (functions). Lately, I've been putting test programs in the packages they test & leaving them there even for the production code.
Once you find the location for some test programs, import the symbol CYBERTIGGYR-TEST:DEFTEST.
You might do this with IMPORT or with the :IMPORT-FROM or :USE options of DEFPACKAGE . Or you might choose not to import the DEFTEST symbol & always qualify it with the CYBERTIGGYR-TEST package name. Whatever floats your boat.
To define a test, use DEFTEST. It works just like DEFUN .
To run tests, evaluate (cybertiggyr-test:run). This will execute all the tests you have defined in any package, not just the current one, until all tests have passwed or some test fails. It will print progress & error messages to *STANDARD-OUTPUT* .

6 Rules for tests

This is the basic contract to which a test must adhere.

A test is a function of no arguments.
A test is a function for which CYBERTIGGYR-TEST:TEST-FUNCTION-P returns true. In other words:
1. It does not have a CYBERTIGGYR-TEST:DISPOSITION property with a value of CYBERTIGGYR-TEST:NOT-A-UNIT-TEST, and
2. It's name³ begins with the characters ``TEST'', or it has a property CYBERTIGGYR-TEST:DISPOSITION with a value of CYBERTIGGYR-TEST:IS-A-UNIT-TEST.
A test returns NIL if & only if it fails. If the test succeeds, it returns true.⁴
The test framework will print the fact that a test passes or fails, but if a test should print a diagnostic message about what went wrong, the test itself must print that message.
The function CYBERTIGGYR-TEST:RUN has control over the order in which tests are run.
It is not necessary to export a test from its package. The test may be bound to a private symbol.

7 Suggestions for tests

A test should print nothing if it succeeds.
When you have a lot of tests, it is not feasible for every test to print a comprehensive performance & statistics dump which you are expected to read to determine whether the test succeeded for failed. When you have a lot of tests, you don't care about statistics; you just want to know what test failed. The RUN function from CYBERTIGGYR-TEST prints the name of each test as it executes the test, & that's all you need or want when the test succeeds. So a test prints nothing if it succeeds.
I've been placing test functions in the packages they test. I do not remove the tests from production code. They end up compiled into the production binaries we deliver.
I number the test functions, like this: TEST0000, TEST0010, ...TEST0101. I do the same thing for test programs in other languages.
At first, it might seems like the name of a test program should indicate what the test tests. In practice, you have so many test programs that making a unique, descriptive name for each is difficult. If a test passes, it doesn't matter what it's called, & if it fails, you'll need to look at its source code, so again, it doesn't matter what it's called.
Use the CHECK macro as an easy way to evaluate an expression & print it if it fails. Here's an example:
```
    (and (check (= (woowoo nil) 1))
         (check (= (woowoo '(1 2 3)) 42)))
```

8 Expert-level details

To tell CYBERTIGGYR-TEST that a particular function is not a test in spite of its name, give it a DISPOSITION of NOT-A-TEST. For example, to ensure TEST-HOOEY is not a test function, you would evaluate ``(setf (get 'test-hooey 'cybertiggyr-test:disposition) 'cybertiggyr-test:not-a-unit-test)''.
Conversely, to make CYBERTIGGYR-TEST treat a function as a test in spite of the function's name, give it a DISPOSITION of IS-A-UNIT-TEST, like this: ``(setf (get 'foo 'cybertiggyr-test:disposition) 'cybertiggyr-test:is-a-unit-test)''.
If you use DEFTEST to define your test functions, you won't need to do this.
To exclude all functions in a package from being treated as tests, push the package object⁵ onto the list CYBERTIGGYR-TEST::*EXCLUDED-PACKAGES*.⁶
*EXCLUDED-PACKAGES* is initialized to include SYSTEM & COMMON-LISP.
By default, functions whose names begin with ``TEST'' are treated as tests. The prefix ``TEST'' isn't hard-coded; it's in the variable CYBERTIGGYR-TEST::*PREFIX*. You can change the ``TEST'' prefix by binding another prefix to CYBERTIGGYR-TEST::*PREFIX*.
Function CYBERTIGGYR-TEST::TEST-FUNCTION-P determines whether a symbol is bound to a function that should be executed as a test. It uses the afore-mentioned *EXCLUDED-PACKAGES* & *PREFIX* variables and the DISPOSITION property.
You can use DEFUN to define tests if the function's name begins with ``TEST'' or you give it a DISPOSITION property of IS-A-UNIT-TEST.
Using DEFTEST is safer, though, because it as an API is independant of how functions are identified as tests.

9 History

I think I first wrote CyberTiggyr Test some time in about 2000, though it could have been as late as 2001 October when I wrote a genetic algorithm library called Evie. It went through a re-write which changed the API in 2002 or 2003. The current version is 3, & its API is backward-compatible with the previous one.

A. Performance Testing

The ``test'' in ``performance testing'' is a misnomer, but I've included a couple of functions for doing it in CyberTiggyr Test. They are in CyberTiggyr Test for convenience.

The RATE function tells how fast a function runs. Call it with the function whose rate you want to know; the function must require no arguments. RATE returns a list of three elements. The list's FIRST is calls per second, it's SECOND is the number of times it was called, & it's THIRD is the number of seconds the test ran. All three numbers are positive, but they may be integers, ratios, or floating point depending on details of your Lisp.
The RATETABLE function runs RATE for a bunch of functions & writes the results to a stream as a LaTeX table. RATETABLE requires one argument, which is a list. Each element in the list has two elements. It's FIRST is the name of the function to time, & it's SECOND is the functio to time.

B. Possible Improvements

Here are some ideas for improvements.

Give user control over the order in which tests are run. Maybe a dependency scheme? Whatever the solution, must be ignorable when user does not care about the order in which tests run.
Export *EXCLUDED-PACKAGES* & *PREFIX*???
Should *PREFIX* be a list of prefixes, not just a single prefix?

C. Other File Formats

This document is available in multi-file HTML format at http://cybertiggyr.com/gene/lut/.
This document is available in Pointless Document Format (PDF) at http://cybertiggyr.com/gene/lut/lut.pdf.

I write almost all of my documents in L^ATEX ([5], [3]). I compile to PDF with latex, dvips, & ps2pdf. I compile to HTML with latex2html ([1], [4]).

Bibliography

1: Nikos Drakos.
latex2html.
2: Free Software Foundation.
Lesser general public license.
world wide web.
http://www.gnu.org/licenses/licenses.html#LGPL.
3: Michel Goossens and Frank Mittelbach.
The L^ATEX Companion.
Addison Wesley Longman, Inc., 1993.
ISBN 0201541998.
4: Michel Goossens and Sebastian Rahtz.
The L^ATEX Web Companion: Integrating TEX, HTML, and XML.
Addison Wesley Longman, Inc., 1999.
ISBN 020143317.
5: Leslie Lamport.
L^ATEX: A Document Preparation System.
Addison-Wesley Publishing Company, Inc., 1986.
ISBN 0-201-15790-X.

Gene Michael Stover 2008-04-20