We will use an established and well-designed regression testing
framework called Prove. It is
not the only possibility though,
FiveAM is a popular one (see
this blogpost for an
introduction) and there are others (and more again). We prefer
Prove
for its documentation and its extensible reporters (it has different
report styles and we can extend them).
warning: Prove has a couple limitations and will soon be obsolete. We advise to start with another test framework.
Testing with Prove
Install and load
Prove
is in Quicklisp:
(ql:quickload "prove")
This command installs prove
if necessary, and loads it.
Write a test file
(in-package :cl-user)
(defpackage my-test
(:use :cl
:prove))
(in-package :my-test)
(subtest "Showing off Prove"
(ok (not (find 4 '(1 2 3))))
(is 4 4)
(isnt 1 #\1))
Prove’s API contains the following testing functions: ok
, is
,
isnt
, is-values
, is-type
, like
(for regexps), is-print
(checks the standard output), is-error
, is-expand
, pass
, fail
,
skip
, subtest
.
Run a test file
(prove:run #P"myapp/tests/my-test.lisp")
(prove:run #P"myapp/tests/my-test.lisp" :reporter :list)
We get an output like:
Run one test
You can directly run one test by compiling it. With Slime, use the
usual C-c C-c
.
More about Prove
Prove
can also:
- be run on Travis CI,
- colorize the output,
- report tests duration,
- change the default test function,
- set a threshold for slow tests,
- invoke the CL debugger whenever getting an error during running tests,
- integrate with ASDF so than we can execute
(asdf:test-system)
or(prove:run)
in the REPL (such configuration is provided by cl-project, by the same author).
Interactively fixing unit tests
Common Lisp is interactive by nature (or so are most implementations), and testing frameworks make use of it. It is possible to ask the framework to open the debugger on a failing test, so that we can inspect the stack trace and go to the erroneous line instantly, fix it and re-run the test from where it left off, by choosing the suggested restart.
With Prove, set prove:*debug-on-error*
to t
.
Below is a short screencast showing all this in action (with FiveAM):
Note that in the debugger:
<enter>
on a backtrace shows more of itv
on a backtrace goes to the corresponding line or function.- you can discover more options with the menu.
Code coverage
A code coverage tool produces a visual output that allows to see what parts of our code were tested or not:
Such capabilities are included into Lisp implementations. For example, SBCL has the sb-cover module and the feature is also built-in in CCL or LispWorks.
Generating an html test coverage output
Let’s do it with SBCL’s sb-cover.
Coverage reports are only generated for code compiled using
compile-file
with the value of the sb-cover:store-coverage-data
optimization quality set to 3.
;;; Load SB-COVER
(require :sb-cover)
;;; Turn on generation of code coverage instrumentation in the compiler
(declaim (optimize sb-cover:store-coverage-data))
;;; Load some code, ensuring that it's recompiled with the new optimization
;;; policy.
(asdf:oos 'asdf:load-op :cl-ppcre-test :force t)
;;; Run the test suite.
(prove:run :yoursystem-test)
Produce a coverage report, set the output directory:
(sb-cover:report "coverage/")
Finally, turn off instrumentation:
(declaim (optimize (sb-cover:store-coverage-data 0)))
You can open your browser at
../yourproject/t/coverage/cover-index.html
to see the report like
the capture above or like
this code coverage of cl-ppcre.
Continuous Integration
Travis CI and Coveralls
Travis is a service for running unit tests in the cloud and Coveralls shows you the evolution of coverage over time, and also tells you what a pull request will do to coverage.
Thanks to cl-travis
we can easily test our program against one or many
Lisp implementations (ABCL, Allegro CL, SBCL, CMUCL, CCL and
ECL). cl-coveralls
helps to post our coverage to the service. It
supports SBCL and Clozure CL with Travis CI and Circle CI.
We refer you to the lengthy and illustrated explanations of the “continuous integration” page on lisp-lang.org.
You’ll find many example projects using them in the links above, but if you want a quick overview of what it looks like:
Gitlab CI
Gitlab CI is part of
Gitlab and is available on Gitlab.com, for
public and private repositories. Let’s see straight away a simple
.gitlab-ci.yml
:
variables:
QUICKLISP_ADD_TO_INIT_FILE: "true"
image: clfoundation/sbcl:latest
before_script:
- install-quicklisp
- git clone https://github.com/foo/bar ~/quicklisp/local-projects/
test:
script:
- make test
Gitlab CI is based on Docker. With image
we tell it to use the latest
tag
of the clfoundation/sbcl
image. This includes the latest version of SBCL, many OS packages useful for CI
purposes, and a script to install Quicklisp. Gitlab will load the image, clone
our project and put us at the project root with administrative rights to run
the rest of the commands.
test
is a “job” we define, script
is a
recognized keywords that takes a list of commands to run.
Suppose we must install dependencies before running our tests: before_script
will run before each job. Here we install Quicklisp (adding it to SBCL’s init
file), and clone a library where Quicklisp can find it.
We can try locally ourselves. If we already installed Docker and
started its daemon (sudo service docker start
), we can do:
docker run --rm -it -v /path/to/local/code:/usr/local/share/common-lisp/source clfoundation/sbcl:latest bash
This will download the lisp image (±300MB compressed), mount some local code in
the image where indicated, and drop us in bash. Now we can try a make test
.
Here is a more complete example that tests against several CL implementations in parallel:
variables:
IMAGE_TAG: latest
QUICKLISP_ADD_TO_INIT_FILE: "true"
QUICKLISP_DIST_VERSION: latest
image: clfoundation/$LISP:$IMAGE_TAG
stages:
- test
- build
before_script:
- install-quicklisp
- git clone https://github.com/foo/bar ~/quicklisp/local-projects/
.test:
stage: test
script:
- make test
abcl test:
extends: .test
variables:
LISP: abcl
ccl test:
extends: .test
variables:
LISP: ccl
ecl test:
extends: .test
variables:
LISP: ecl
sbcl test:
extends: .test
variables:
LISP: sbcl
build:
stage: build
variables:
LISP: sbcl
only:
- tags
script:
- make build
artifacts:
paths:
- some-file-name
Here we defined two stages
(see
environments),
“test” and “build”, defined to run one after another. A “build” stage
will start only if the “test” one succeeds.
“build” is asked to run only
when a
new tag is pushed, not at every commit. When it succeeds, it will make
the files listed in artifacts
’s paths
available for download. We can
download them from Gitlab’s Pipelines UI, or with an url. This one will download
the file “some-file-name” from the latest “build” job:
https://gitlab.com/username/project-name/-/jobs/artifacts/master/raw/some-file-name?job=build
When the pipelines pass, you will see:
You now have a ready to use Gitlab CI.
Page source: testing.md