So you want to easily test the code you’re writing? The following recipes cover how to write automated tests and see their code coverage. We also give pointers to plug those in modern continuous integration services like Travis CI and Coveralls.

We will use an established and well-designed regression testing framework called Prove. It is not the only possibility though, FiveAM is a popular one (see this blogpost for an introduction) and there are others (and more again). We prefer Prove for its documentation and its extensible reporters (it has different report styles and we can extend them).

warning: Prove has a couple limitations and will soon be obsolete. We advise to start with another test framework, such as FiveAM.

Testing with Prove

Install and load

Prove is in Quicklisp:

(ql:quickload "prove")

This command installs prove if necessary, and loads it.

Write a test file

(in-package :cl-user)
(defpackage my-test
  (:use :cl
        :prove))
(in-package :my-test)

(subtest "Showing off Prove"
  (ok (not (find 4 '(1 2 3))))
  (is 4 4)
  (isnt 1 #\1))

Prove’s API contains the following testing functions: ok, is, isnt, is-values, is-type, like (for regexps), is-print (checks the standard output), is-error, is-expand, pass, fail, skip, subtest.

Run a test file

(prove:run #P"myapp/tests/my-test.lisp")
(prove:run #P"myapp/tests/my-test.lisp" :reporter :list)

We get an output like:

Run one test

You can directly run one test by compiling it. With Slime, use the usual C-c C-c.

More about Prove

Prove can also:

be run on Travis CI,
colorize the output,
report tests duration,
change the default test function,
set a threshold for slow tests,
invoke the CL debugger whenever getting an error during running tests,
integrate with ASDF so than we can execute (asdf:test-system) or (prove:run) in the REPL (such configuration is provided by cl-project, by the same author).

See Prove’s documentation!

Interactively fixing unit tests

Common Lisp is interactive by nature (or so are most implementations), and testing frameworks make use of it. It is possible to ask the framework to open the debugger on a failing test, so that we can inspect the stack trace and go to the erroneous line instantly, fix it and re-run the test from where it left off, by choosing the suggested restart.

With Prove, set prove:*debug-on-error* to t.

Below is a short screencast showing all this in action (with FiveAM):

Note that in the debugger:

<enter> on a backtrace shows more of it
v on a backtrace goes to the corresponding line or function.
you can discover more options with the menu.

Code coverage

A code coverage tool produces a visual output that allows to see what parts of our code were tested or not:

Such capabilities are included into Lisp implementations. For example, SBCL has the sb-cover module and the feature is also built-in in CCL or LispWorks.

Generating an html test coverage output

Let’s do it with SBCL’s sb-cover.

Coverage reports are only generated for code compiled using compile-file with the value of the sb-cover:store-coverage-data optimization quality set to 3.

;;; Load SB-COVER
(require :sb-cover)

;;; Turn on generation of code coverage instrumentation in the compiler
(declaim (optimize sb-cover:store-coverage-data))

;;; Load some code, ensuring that it's recompiled with the new optimization
;;; policy.
(asdf:oos 'asdf:load-op :cl-ppcre-test :force t)

;;; Run the test suite.
(prove:run :yoursystem-test)

Produce a coverage report, set the output directory:

(sb-cover:report "coverage/")

Finally, turn off instrumentation:

(declaim (optimize (sb-cover:store-coverage-data 0)))

You can open your browser at ../yourproject/t/coverage/cover-index.html to see the report like the capture above or like this code coverage of cl-ppcre.

Continuous Integration

Continuous Integration is important to run automatic tests after a commit or before a pull request, to run code quality checks, to build and distribute your software… well, to automate everything about software.

We want our programs to be portable across Lisp implementations, so we’ll set up our CI pipeline to run our tests against several of them (it could be SBCL and CCL of course, but while we’re at it ABCL, ECL and possibly more).

We have a choice of Continuous Integration services: Travis CI, Circle, Gitlab CI, now also GitHub Actions, etc (many existed before GitHub Actions, if you wonder). We’ll have a look at how to configure a CI pipeline for Common Lisp, and we’ll focus a little more on Gitlab CI on the last part.

We’ll also quickly show how to publish coverage reports to the Coveralls service. cl-coveralls helps to post our coverage to the service.

GitHub Actions, Circle CI, Travis… with CI-Utils

We’ll use CI-Utils, a set of utilities that comes with many examples. It also explains more precisely what is a CI system and compares a dozen of services.

It relies on Roswell to install the Lisp implementations and to run the tests. They all are installed with a bash one-liner:

curl -L https://raw.githubusercontent.com/roswell/roswell/release/scripts/install-for-ci.sh | bash

(note that on the Gitlab CI example, we use a ready-to-use Docker image that contains them all)

It also ships with a test runner for FiveAM, which eases some rough parts (like returning the right error code to the terminal). We install ci-utils with Roswell, and we get the run-fiveam executable.

Then we can run our tests:

run-fiveam -e t -l foo/test :foo-tests  # foo is our project

Following is the complete .travis.yml file.

The first part should be self-explanatory:

### Example configuration for Travis CI ###
language: generic

addons:
  homebrew:
    update: true
    packages:
    - roswell
  apt:
    packages:
      - libc6-i386 # needed for a couple implementations
      - default-jre # needed for abcl

# Runs each lisp implementation on each of the listed OS
os:
  - linux
#  - osx # OSX has a long setup on travis, so it's likely easier to just run select implementations on OSX

This is how we configure the implementations matrix, to run our tests on several Lisp implementations. We also send the test coverage made with SBCL to Coveralls.

env:
  global:
    - PATH=~/.roswell/bin:$PATH
    - ROSWELL_INSTALL_DIR=$HOME/.roswell
#    - COVERAGE_EXCLUDE=t  # for prove or rove
  jobs:
    # The implementation and whether coverage is send to coveralls are controlled with these environmental variables
    - LISP=sbcl-bin COVERALLS=true
    - LISP=ccl-bin
    - LISP=abcl
    - LISP=ecl   # warn: in our experience, compilations times can be long on ECL.

# Additional OS/Lisp combinations can be added to those generated above
jobs:
  include:
    - os: osx
      env: LISP=sbcl-bin
    - os: osx
      env: LISP=ccl-bin

Some jobs can be marked as allowed to fail:

# Note that this should only be used if there is no interest for the library to work on that system
#  allow_failures:
#    - env: LISP=abcl
#    - env: LISP=ecl
#    - env: LISP=cmucl
#    - env: LISP=alisp
#      os: osx

  fast_finish: true

We finally install Roswell, the implementations, and we run our tests.

cache:
  directories:
    - $HOME/.roswell
    - $HOME/.config/common-lisp

install:
  - curl -L https://raw.githubusercontent.com/roswell/roswell/release/scripts/install-for-ci.sh | sh
  - ros install ci-utils #for run-fiveam
#  - ros install prove #for run-prove
#  - ros install rove #for [run-] rove

  # If asdf 3.16 or higher is needed, uncomment the following lines
  #- mkdir -p ~/common-lisp
  #- if [ "$LISP" == "ccl-bin" ]; then git clone https://gitlab.common-lisp.net/asdf/asdf.git ~/common-lisp; fi

script:
  - run-fiveam -e t -l foo/test :foo-tests
  #- run-prove foo.asd
  #- rove foo.asd

Below with Gitlab CI, we’ll use a Docker image that already contains the Lisp binaries and every Debian package required to build Quicklisp libraries.

Gitlab CI

Gitlab CI is part of Gitlab and is available on Gitlab.com, for public and private repositories. Let’s see straight away a simple .gitlab-ci.yml:

variables:
  QUICKLISP_ADD_TO_INIT_FILE: "true"

image: clfoundation/sbcl:latest

before_script:
  - install-quicklisp
  - git clone https://github.com/foo/bar ~/quicklisp/local-projects/

test:
  script:
    - make test

Gitlab CI is based on Docker. With image we tell it to use the latest tag of the clfoundation/sbcl image. This includes the latest version of SBCL, many OS packages useful for CI purposes, and a script to install Quicklisp. Gitlab will load the image, clone our project and put us at the project root with administrative rights to run the rest of the commands.

test is a “job” we define, script is a recognized keywords that takes a list of commands to run.

Suppose we must install dependencies before running our tests: before_script will run before each job. Here we install Quicklisp (adding it to SBCL’s init file), and clone a library where Quicklisp can find it.

We can try locally ourselves. If we already installed Docker and started its daemon (sudo service docker start), we can do:

docker run --rm -it -v /path/to/local/code:/usr/local/share/common-lisp/source clfoundation/sbcl:latest bash

This will download the lisp image (±300MB compressed), mount some local code in the image where indicated, and drop us in bash. Now we can try a make test.

Here is a more complete example that tests against several CL implementations in parallel:

variables:
  IMAGE_TAG: latest
  QUICKLISP_ADD_TO_INIT_FILE: "true"
  QUICKLISP_DIST_VERSION: latest

image: clfoundation/$LISP:$IMAGE_TAG

stages:
  - test
  - build

before_script:
  - install-quicklisp
  - git clone https://github.com/foo/bar ~/quicklisp/local-projects/

.test:
  stage: test
  script:
    - make test

abcl test:
  extends: .test
  variables:
    LISP: abcl

ccl test:
  extends: .test
  variables:
    LISP: ccl

ecl test:
  extends: .test
  variables:
    LISP: ecl

sbcl test:
  extends: .test
  variables:
    LISP: sbcl

build:
  stage: build
  variables:
    LISP: sbcl
  only:
    - tags
  script:
    - make build
  artifacts:
    paths:
      - some-file-name

Here we defined two stages (see environments), “test” and “build”, defined to run one after another. A “build” stage will start only if the “test” one succeeds.

“build” is asked to run only when a new tag is pushed, not at every commit. When it succeeds, it will make the files listed in artifacts’s paths available for download. We can download them from Gitlab’s Pipelines UI, or with an url. This one will download the file “some-file-name” from the latest “build” job:

https://gitlab.com/username/project-name/-/jobs/artifacts/master/raw/some-file-name?job=build

When the pipelines pass, you will see:

You now have a ready to use Gitlab CI.

References

the CL Foundation Docker images

Page source: testing.md

The Common Lisp Cookbook – Testing