(sph test)


test definition

  • tests can be grouped into composable modules
  • simple syntax for input/output arguments
  • nestable assertions with titles that get concatenated
  • hooks for custom before/after/inbetween procedures and custom reporters
  • using test-modules is optional


  • all modules and procedures for testing are compiled before test procedures are run, so that no auto-compile messages or compile errors appear while testing
  • test-modules can extend or customise main test runner settings
  • no restriction on how modules will run tests. for example, modules can run tests "repeated three times, in parallel"
  • tests in multiple files can be run as one, with grouped output
  • "only" and "exclude" options for selecting specific tests or modules to be run


  • scheme interface - execute tests from within scheme, for continuous integration for example
  • standard-output result reporting is optional
  • space efficient reporting format "compact"
  • extendable report formats, stored in a hashtable and selected via test settings


tests are procedures, test-modules are r6rs-libraries, mostly functional, no hidden state variables, no goops objects, no opaque data-types,

assert bindings

all assert bindings accept an optional title/name string as the first argument, which is eventually displayed and stored in the result. on failure, the titles of nested asserts get combined to create the name of the test.

(assert-true (= 3 (+ 2 1)))
(assert-true "addition-test" (= 3 (+ 2 1)))
(assert-equal "addition-test" (+ 1 2) 3)
(assert-equal (+ 1 2) 3)
(assert-and "groupname"
  (assert-true "addition-test" (= 3 (+ 2 1)))
  (assert-equal "another addition-test" (= 6 (+ 5 4)))
  (assert-true (+ 2 1)))


file content

(define-test-module (test module sph string)
  (import (sph string))

      ("/a////b//c/d" "/") (0 2 3 4 5 7 8 10)
      ("/a////b//c/d" "//") (2 4 7)
      ("abcd" "bc") (1)
      ("abcdbcefbcg" "bc") (1 4 8)
      ("abcd" "cd") (2)
      ("ab" "") (0 1 2)
      ("" "") (0)
      ("" "ab") ()
      ("abcd" "cb") ())
      ("a" 0) ""
      ("a" 3) "aaa"
      ("" 3) "")
      ("/a////b//c/d" "//" "/") "/a//b/c/d"
      ("abcd" "bc" "") "ad"
      ("abcdbcefbcg" "bc" "hij") "ahijdhijefhijg"
      ("abcd" "abc" "e") "ed"
      ("abcd" "cd" "efg") "abefg"
      ("ab" "" "") "ab"
      ("ab" "" "/") "/a/b/"
      ("" "" "/") "/"
      ("" "" "") "")))

test-execute-procedures-lambda ist a helper for defining basic input/output tests.

report in the compact format

string-indices 1 2 3 4 5 6 7 8 9
string-multiply 1 2 3
string-replace-string 1 2 3 4 5 6 7 8 9

report with failure

  • inp: the arguments to the test-procedure
  • exp: the expected result
  • out: the actual result
string-indices 1 2 3 4 5 6 7 8 9
string-multiply 1 2 3
string-replace-string 1 2 3 4 5 6 7 8 9
string-quote 1
  failure string-quote 2
  inp "t'est"
  exp "\"t'est\"x"
  out "\"t'est\""

report with multiple modules

test module sph vector
  vector-append 1
  vector-produce 1
  vector-range-ref 1 2 3
  vector-select 1
test module sph record
  define-record 1
  record 1
  alist->record 1
  record-field-names 1

recommended module directory structure for tests

    custom-test-module-files ...
    custom-test-helper-module-files ...


test-module definition

(define-test-module (symbol ...)
  (import r6rs-library-import-spec ...)
  any ...
  procedure:{settings:list -> boolean/vector:test-result})

the following libraries are implicitly imported: (guile) (rnrs base) (sph) (sph test). the last expression of a test-module must be a procedure. this procedure will be executed to get the test result for the module.

(define-test-module (test module mymodule)
  (import (mymodule))

  (lambda (settings)
    ;create a test-result, a boolean would do it, but the following creates a test-result record with more information
    (assert-true "addition-test" (= 3 (+ 2 1)))))

the rest of the define-test-module body is exactly like the body of a r6rs-library definition. another example of how a module can look like:

(define-test-module (test module my-module)
  (import (my-module))

  (define (my-utility a b) (+ a b))

  (define-test (addition)
    (assert-true "addition-test" (= 3 (+ 2 1))))

  (define-test (addition-2 arguments)
    (apply + arguments))

    (addition-2 (5 4) 9)
      (1 2) 3
      (3 4) 7)))

test procedure execution

test-results in a test-modules execute procedure can be created in any way scheme permits. (sph test) offers a few additional bindings for that. they all use a specific notation for tests to be run, the test arguments and expected-results, called "test-spec".


symbol/(symbol [any/(any ...) ...])
procedure-name/(procedure-name [arguments-and-expected-result-alternatingly ...])


(string-quote "'test'" "\"'test'\"")
(string-multiply ("" 3) "")
  • test procedures are called once for each arguments/expected-result pairing
  • procedure-name is without any "test-" prefix. the procedure-name is looked up with a "test-" prefix first, but if not found, the procedure-name without the prefix is used
  • the test procedures arguments, or the arguments for the procedure to be tested, and expected results are specified alternatingly. this makes it simple to create basic input/output tests
  • lists for arguments are passed as multiple arguments, but lists for expected results are not considered to be multiple return values


test-execute-procedures-lambda :: test-spec ...

can be used as the last expression in a test-module definition. creates a anonymous procedure that takes and uses a settings argument and evaluates test-specs in an implicit quasiquote.

(test-execute-procedures-lambda (my-procedure (unquote (+ 1 2)) 3))

define-procedure-tests :: binding-name test-spec ...

define a variable with a list of test-specs like test-execute-procedures-lambda

test-execute-procedures :: list:settings list:tests

runs tests test procedures are not required to return test-result records, any other value is compared against the specified expected output

test procedure definition

"test-execute-procedures" and similar bindings look for custom test procedures with certain type signatures. the following syntax can be used to define such procedures.

define test

(define-test (name argument-name ...) test-result)


(define-test name procedure)
(define-test (abc) #t)
(define-test (abc arguments-list) #t)
(define-test (abc arguments-list expected-result) #t)
(define-test (abc name index arguments-list expected-result) #t)

both or none of the two name/index arguments need to be specified.. internally, test procedures are top-level bindings with a name that is prefixed with "test-".

fundamental type signature

test-{name} :: [name index] arguments expected-result -> any/vector/boolean:test-result
(define (test-abc) #t)
(define (test-abc arguments-list) #t)
(define (test-abc arguments-list expected-result) #t)
(define (test-abc name index arguments-list expected-result) #t)

test-module execution


  • test-execute-module :: list:settings (symbol ...):module-name -> test-result
  • test-execute-modules :: list:settings ((symbol ...) ...):module-names -> test-result
  • test-execute-modules-by-prefix :: #:settings list (symbol ...):module-name-part -> test-result

data structure

test-result: ([group-name] test-result ...)/test-result-record



(import (sph) (sph test))
(define settings (test-settings-default-custom path-search "modules" reporter-name (q compact)))
(test-execute-modules-by-prefix #:settings settings (q (test module sph)))

create custom test-results

  • module test results can be booleans indicating success status
  • test-execute-procedures results can be booleans or any other result to be compared with the expected result
  • alternatively any result can be a test-result record with additional information
(test-create-result [boolean:success? string:title integer:index any:result list:arguments any:expected])

customise settings, hooks and reporters


  • settings are stored as association lists
  • since the modification of alists (or any dictionary data structure) is not necessarily straightforward in basic scheme, the binding "alist-q-merge-key/value" from (sph alist) is used to update values in an association list

following is an example of a test-module execute procedure for updating settings and installing a hook that runs before each new procedure test

(define-test-module (test module example)
  (import (sph test) (sph alist) (sph storage dg) (test helper sph storage dg))

  (define-procedure-tests tests
      123456 "files/1e240"
      789 "files/315"))

  (lambda (settings)
        (alist-q-merge-key/value settings
          random-order? #t
          (alist-q-merge-key/value (alist-q-ref settings hook)
            (lambda a
                (not (dg-index-no-errors? (or (dg-index-errors-intern) (dg-index-errors-pair))))
                (throw (q index-corruption)))))))
      (test-execute-procedures settings tests))))

test helpers

given the recommended directory structure for tests, one can create standard r6rs libraries in files stored in a load-path under test/helper/ and import them in test-modules

test environment/data/fixtures

currently it is fully up to the user to create and destroy eventually necessary environments for tests. for example database initialisation with test data. given the hook system of (sph test), it should be possible to create nice abstractions for that.


settings may apply to procedure, module and modules tests. the following examples use the "alist-q" binding from (sph alist), which creates an alist from an alternated key/value specification where keys are quoted. "define-as" is also used, which is the same as "(define test-settings-default (alist-q _ ...))".

default settings

(define-as test-settings-default alist-q
  reporters test-reporters-default
  reporter-name (q default)
  search-type (q prefix)
  (alist-q procedure-before ignore
    procedure-after ignore
    procedure-data-before ignore
    procedure-data-after ignore
    module-before ignore module-after ignore modules-before ignore modules-after ignore)
  random-order? #f parallel? #f exception-strings? #t exclude #f only #f until #f)

exception-strings?: boolean

if true, exeptions occuring in test-execute-procedures tests are catched and converted to strings that will be the result of the test procedure. this way, tests for if an exception occurs can be made.

exclude: (symbol:test-procedure-name/(symbol ...):module-name ...)

list of procedure or module names to exclude from test execution

hook: list:hook-configuration

the association list stored under the "hook" key contains procedures that are called at specific times in the test execution process. the type signatures can be seen in the following scheme comments.

(define-as test-report-hooks-null alist-q
  ;settings name ->
  procedure-before ignore
  ;settings result ->
  procedure-after ignore
  ;settings name index data ->
  procedure-data-before ignore
  ;settings result ->
  procedure-data-after ignore
  ;settings module-name ->
  module-before ignore
  ;settings module-name result ->
  module-after ignore
  ;settings module-names ->
  modules-before ignore
  ;settings module-names result ->
  modules-after ignore)

only: (symbol:test-procedure-name/(symbol ...):module-name ...)

  • takes preference over "exclude"
  • either "only" or "exclude" are applied, not both
  • "only" and "exclude" is used for both module and procedure tests

parallel?: boolean

  • tests are executed in separate threads
  • all tests will be tried to be executed even if one fails

path-search: string

  • a load path (a root path from which module names start) to restrict the search for modules to
  • one use case is to execute only tests local to a project directory
  • this can be used to execute test-modules that have not been installed into a guile load path, without conflicting with modules that are installed there

random-order?: boolean

the test order is randomised before execution

reporter-name: symbol

default reporter names are "compact", "null" and "default" (which is equivalent to "null")

reporters: hashtable

symbol:name -> (procedure:report-result:{test-result [port] ->} . list:hook-configuration)

reporter configuration is a pair that consists of a procedure for reporting a test-result object and a hook-configuration

search-type: symbol:exact/prefix/prefix-not-exact

specifies how to match module names: partially (prefix or prefix-not-exact) and therefore possibly multiple modules, or as exact, full module names

until: symbol:test-procedure-name/(symbol ...):module-name

stops when reaching the specified procedure/module



  • modules are r6rs-libraries that export only one procedure named "execute" that is supposed to create a test-result
  • execute :: settings -> test-result
  • modules are resolved in the scheme implementation specific way
  • libraries are used to give test-modules a separated scope to include the modules they depend on
  • it also gives test-modules a resolvable name. an alternative implementation could have evaluated top-level file content, which might use "import", in a custom environment, but this might lead to less separation of the module content. test-modules can be imported like other libraries as long as the define-test-module syntax is available before import
  • to use the define-test-module syntax, the define-test-module syntax definition must be previously loaded in the environment in which the module is to be loaded. to archieve this, module file content is first evaluated in the top-level environment to define the module, then the module object is resolved
  • modules export the procedure to accept the settings binding and to delay execution


  • list/record/boolean
  • procedure output is compared with the expected value to determine test success
  • the arguments and expected value are only set on failure
  • procedure tests with multiple test-data create only one test-result with the index set to the last tested index


this macro exists for leaving out optional parameters and to avoid the need to manually prepend "test-" to the name.


  • continuation passing style for procedure tests: calling procedure tests with a "done" procedure. this was considered, but no use case has been found. threads and futures are available for asynchronous processing
  • terminal output coloring. too subjective, can be distracting, not necessary, color escape sequences can make trouble with output processing
  • "fun" reporters (nyan cat, plane landing, etc)
  • implicit multiple return values support similar to how multiple input arguments are specified in a list. would require that every specified expected result value is wrapped in a list

possible enhancements

  • counting tests. when executing many tests, it is currently not displayed how many tests will be executed
  • reporters for csv and scm formats
  • define-test-module form that doesnt wrap content
  • testing and finishing of the test-module composition feature: creating test-modules that execute other test-modules. the library has been designed with this in mind, but it has not really been tested

previously existing work

in other languages

  • mocha. defines test procedures and data inline
  • trc-testing
  • rails testing

other scheme test libraries