2017-10-19

(sph test)

automated code testing with composable modules.

part of sph-lib

features

definition

tests can be grouped into composable modules

simple syntax for input/output arguments

nestable assertions with titles that get concatenated

hooks for custom before/after/inbetween procedures and custom reporters

test-modules are optional

execution

modules and procedures for testing are pre-compiled before test procedures run so that no auto-compile messages or compile errors appear while testing

test-modules can extend/customise test runner settings

no restriction on how modules run tests, for example modules can run tests "repeated three times, in parallel"

tests in files or directories can be run combined

"only"/"exclude" options for selecting tests or modules

runs faster than the previous (sph test) version

interface

scheme interface - execute tests from within scheme for continuous integration for example

standard-output result reporting is optional

space efficient reporting format "compact"

extendable report formats, stored in a hashtable and selected via test settings

does not define procedures with names that start with %

implementation

no opaque data-types, no goops objects, tests are procedures, test-modules are r6rs-libraries

mostly functional, no hidden state variables

assert bindings

all assert bindings accept an optional test title/name string as the first argument which is eventually displayed and stored in the result

on failure, the titles of nested asserts get combined to create the name of the test

(assert-true (= 3 (+ 2 1)))
(assert-true "addition-test" (= 3 (+ 2 1)))
(assert-equal "addition-test" (+ 1 2) 3)
(assert-equal (+ 1 2) 3)
(assert-and "groupname"
  (assert-true "addition-test" (= 3 (+ 2 1)))
  (assert-equal "another addition-test" (= 6 (+ 5 4)))
  (assert-true (+ 2 1)))

examples

file content

(define-test-module (test module sph string)
  (import (sph string))
  
  (test-execute-procedures-lambda
    (string-indices
      ("/a////b//c/d" "/") (0 2 3 4 5 7 8 10)
      ("/a////b//c/d" "//") (2 4 7)
      ("abcd" "bc") (1)
      ("abcdbcefbcg" "bc") (1 4 8)
      ("abcd" "cd") (2)
      ("ab" "") (0 1 2)
      ("" "") (0)
      ("" "ab") ()
      ("abcd" "cb") ())
    (string-multiply
      ("a" 0) ""
      ("a" 3) "aaa"
      ("" 3) "")
    (string-replace-string
      ("/a////b//c/d" "//" "/") "/a//b/c/d"
      ("abcd" "bc" "") "ad"
      ("abcdbcefbcg" "bc" "hij") "ahijdhijefhijg"
      ("abcd" "abc" "e") "ed"
      ("abcd" "cd" "efg") "abefg"
      ("ab" "" "") "ab"
      ("ab" "" "/") "/a/b/"
      ("" "" "/") "/"
      ("" "" "") "")))

report in the compact format

string-indices 1 2 3 4 5 6 7 8 9
string-multiply 1 2 3
string-replace-string 1 2 3 4 5 6 7 8 9

report with failure

inp: the arguments to the test-procedure

exp: the expected result

out: the actual result

string-indices 1 2 3 4 5 6 7 8 9
string-multiply 1 2 3
string-replace-string 1 2 3 4 5 6 7 8 9
string-quote 1
  failure string-quote 2
  inp "t'est"
  exp "\"t'est\"x"
  out "\"t'est\""

report with multiple modules

test module sph vector
  vector-append 1
  vector-produce 1
  vector-range-ref 1 2 3
  vector-select 1
test module sph record
  define-record 1
  record 1
  alist->record 1
  record-field-names 1

recommended module directory structure for tests

test
  module
    custom-test-module-files ...
  helper
    custom-test-helper-module-files ...

usage

test-module definition

(define-test-module (symbol ...)
  (import r6rs-library-import-spec ...)
  any ...
  procedure:{settings:list -> boolean/vector:test-result})

the following libraries are implicitly imported: (guile) (rnrs base) (sph) (sph test)

the last expression of a test-module must be a procedure. this procedure will be executed to get the test result for the module

(define-test-module (test module mymodule)
  (import (mymodule))
  
  (lambda (settings)
    ;create a test-result, a boolean would do it, but the following creates a test-result record with more information
    (assert-true "addition-test" (= 3 (+ 2 1)))))

the rest of the define-test-module body is exactly like the body of a r6rs-library definition

another example of how a module can look like:

(define-test-module (test module my-module)
  (import (my-module))
  
  (define (my-utility a b) (+ a b))
  
  (define-test (addition)
    (assert-true "addition-test" (= 3 (+ 2 1))))
  
  (define-test (addition-2 arguments)
    (apply + arguments))
  
  (test-execute-procedures-lambda
    addition
    (addition-2 (5 4) 9)
    (+
      (1 2) 3
      (3 4) 7)))

test procedure execution

test-results in a test-modules execute procedure can be created in any way scheme permits. (sph test) offers a few additional bindings for that

they all use a specific notation for tests to be run, the test arguments and expected-results, called "test-spec"

test-spec

symbol/(symbol [any/(any ...) ...])
procedure-name/(procedure-name [arguments-and-expected-result-alternatingly ...])

example

(string-quote "'test'" "\"'test'\"")
(string-multiply ("" 3) "")

test procedures are called once for each arguments/expected-result pairing

procedure-name is without any "test-" prefix. the procedure-name is looked up with a "test-" prefix first, but if not found, the procedure-name without the prefix is used

the test procedures arguments, or the arguments for the procedure to be tested, and expected results are specified alternatingly. this makes it simple to create basic input/output tests

lists for arguments are passed as multiple arguments, but lists for expected results are not considered to be multiple return values

bindings

test-execute-procedures-lambda :: test-spec ...

can be used as the last expression in a test-module definition. creates a anonymous procedure that takes and uses a settings argument and

evaluates test-specs in an implicit quasiquote

(test-execute-procedures-lambda (my-procedure (unquote (+ 1 2)) 3))

define-procedure-tests :: binding-name test-spec ...

define a variable with a list of test-specs like test-execute-procedures-lambda

test-execute-procedures :: list:settings list:tests

runs tests

test procedures are not required to return test-result records, any other value is compared against the specified expected output

test procedure definition

"test-execute-procedures" and similar bindings look for custom test procedures with certain type signatures

the following syntax can be used to define such procedures

define test

(define-test (name argument-name ...) test-result)

examples

(define-test name procedure)
(define-test (abc) #t)
(define-test (abc arguments-list) #t)
(define-test (abc arguments-list expected-result) #t)
(define-test (abc name index arguments-list expected-result) #t)

both or none of the two name/index arguments need to be specified

internally test procedures are top-level bindings with a name that is prefixed with "test-"

fundamental type signature

test-{name} :: [name index] arguments expected-result -> any/vector/boolean:test-result
(define (test-abc) #t)
(define (test-abc arguments-list) #t)
(define (test-abc arguments-list expected-result) #t)
(define (test-abc name index arguments-list expected-result) #t)

test-module execution

bindings

test-execute-module :: list:settings (symbol ...):module-name -> test-result

test-execute-modules :: list:settings ((symbol ...) ...):module-names -> test-result

test-execute-modules-by-prefix :: #:settings list (symbol ...):module-name-part -> test-result

data structure

test-result: ([group-name] test-result ...)/test-result-record

example

#!/usr/bin/guile
!#

(import (sph) (sph test))
(define settings (test-settings-default-custom path-search "modules" reporter-name (q compact)))
(test-execute-modules-by-prefix #:settings settings (q (test module sph)))

create custom test-results

module test results can be booleans indicating success status

test-execute-procedures results can be booleans or any other result to be compared with the expected result

alternatively any result can be a test-result record with additional information

(test-create-result [boolean:success? string:title integer:index any:result list:arguments any:expected])

customise settings, hooks and reporters

settings

settings are stored as association lists

since the modification of alists (or any dictionary data structure) is not necessarily straightforward in basic scheme, the binding "alist-q-merge-key/value" from (sph alist) is used to update values in an association list

following is an example of a test-module execute procedure for updating settings and installing a hook that runs before each new procedure test

(define-test-module (test module example)
  (import (sph test) (sph alist) (sph storage dg) (test helper sph storage dg))
  
  (define-procedure-tests tests
    (dg-create-ide)
    (dg-aliases)
    (dg-ide->path
      123456 "files/1e240"
      789 "files/315"))
  
  (lambda (settings)
    (let
      (settings
        (alist-q-merge-key/value settings
          random-order? #t
          hook
          (alist-q-merge-key/value (alist-q-ref settings hook)
            procedure-before
            (lambda a
              (test-env-dg-reset)
              (if
                (not (dg-index-no-errors? (or (dg-index-errors-intern) (dg-index-errors-pair))))
                (throw (q index-corruption)))))))
      (test-execute-procedures settings tests))))

test helpers

given the recommended directory structure for tests, one can create standard r6rs libraries in files stored in a load-path under test/helper/ and import them in test-modules

test environment/data/fixtures

currently it is fully up to the user to create and destroy eventually necessary environments for tests. for example database initialisation with test data.

given the hook system of (sph test) it should be possible to create nice abstractions for that

settings

settings may apply to procedure, module and modules tests

the following examples use the "alist-q" binding from (sph alist), which creates an alist from an alternated key/value specification where keys are quoted. "define-as" is also used, which is the same as "(define test-settings-default (alist-q _ ...))"

default settings

(define-as test-settings-default alist-q
  reporters test-reporters-default
  reporter-name (q default)
  search-type (q prefix)
  hook
  (alist-q procedure-before ignore
    procedure-after ignore
    procedure-data-before ignore
    procedure-data-after ignore
    module-before ignore module-after ignore modules-before ignore modules-after ignore)
  random-order? #f parallel? #f exception-strings? #t exclude #f only #f until #f)

exception-strings?: boolean

if true, exeptions occuring in test-execute-procedures tests are catched and converted to strings that will be the result of the test procedure

this way tests for if an exception occurs can be made

exclude: (symbol:test-procedure-name/(symbol ...):module-name ...)

list of procedure or module names to exclude from test execution

hook: list:hook-configuration

the association list stored under the "hook" key contains procedures that are called at specific times in the test execution process

the type signatures can be seen in the following scheme comments

(define-as test-report-hooks-null alist-q
  ;settings name ->
  procedure-before ignore
  ;settings result ->
  procedure-after ignore
  ;settings name index data ->
  procedure-data-before ignore
  ;settings result ->
  procedure-data-after ignore
  ;settings module-name ->
  module-before ignore
  ;settings module-name result ->
  module-after ignore
  ;settings module-names ->
  modules-before ignore
  ;settings module-names result ->
  modules-after ignore)

only: (symbol:test-procedure-name/(symbol ...):module-name ...)

takes preference over "exclude"

either "only" or "exclude" are applied, not both

"only" and "exclude" is used for both module and procedure tests

parallel?: boolean

tests are executed in separate threads

all tests will be tried to be executed even if one fails

path-search: string

a load path (a root path from which module names start) to restrict the search for modules to

one use case is to execute only tests local to a project directory

this can be used to execute test-modules that have not been installed into a guile load path, without conflicting with modules that are installed there

random-order?: boolean

the test order is randomised before execution

reporter-name: symbol

default reporter names are "compact", "null" and "default" (which is equivalent to "null")

reporters: hashtable

symbol:name -> (procedure:report-result:{test-result [port] ->} . list:hook-configuration)

reporter configuration is a pair that consists of a procedure for reporting a test-result object and a hook-configuration

search-type: symbol:exact/prefix/prefix-not-exact

specifies how to match module names: partially (prefix or prefix-not-exact) and therefore possibly multiple modules, or as exact, full module names

until: symbol:test-procedure-name/(symbol ...):module-name

stops when reaching the specified procedure/module

implementation

test-modules

modules are r6rs-libraries that export only one procedure named "execute" that is supposed to create a test-result

execute :: settings -> test-result

modules are resolved in the scheme implementation specific way

libraries are used to give test-modules a separated scope to include the modules they depend on

it also gives test-modules a resolvable name. an alternative implementation could have evaluated top-level file content, which might use "import", in a custom environment, but this might lead to less separation of the module content. test-modules can be imported like other libraries as long as the define-test-module syntax is available before import

to use the define-test-module syntax, the define-test-module syntax definition must be previously loaded in the environment in which the module is to be loaded. to archieve this, module file content is first evaluated in the top-level environment to define the module, then the module object is resolved

modules export the procedure to accept the settings binding and to delay execution

test-result

list/record/boolean

procedure output is compared with the expected value to determine test success

the arguments and expected value are only set on failure

procedure tests with multiple test-data create only one test-result with the index set to the last tested index

define-test

this macro exists for leaving out optional parameters and to avoid the need to manually prepend "test-" to the name

excluded

continuation passing style for procedure tests: calling procedure tests with a "done" procedure. this was considered, but no use case has been found. threads and futures are available for asynchronous processing

terminal output coloring. too subjective, can be distracting, not necessary, color escape sequences can make trouble with output processing

"fun" reporters (nyan cat, plane landing, etc)

implicit multiple return values support similar to how multiple input arguments are specified in a list. would require that every specified expected result value is wrapped in a list

possible enhancements

counting tests. when executing many tests, it is currently not displayed how many tests will be executed

reporters for csv and scm formats

testing and finishing of the test-module composition feature: creating test-modules that execute other test-modules. the library has been designed with this in mind, but it has not really been tested

previously existing work

mocha. defines test procedures and data inline

trc-testing

rails testing

other test libraries

library description

# data structures:

test-result: ([group-name] test-result ...)/vector

# syntax

test-settings-default-custom :: [any:unquoted-key any:value] ... -> list

test-list :: symbol:name/(symbol:name any:io-data ...) ... -> ((symbol procedure [arguments expected] ...) ...)

define-test

  :: (name [arguments expected settings]) body ...

  :: name procedure body ...

  define a procedure with the name test-{name} and the arity expected by the default evaluators

define-procedure-tests :: symbol:name symbol/literal-list

  define a variable with tests that can be executed with test-execute-procedures.

  resolves procedures by name and normalises the test specification

test-execute-procedures-lambda :: test-list-arguments ... -> procedure:{settings -> test-result}

  create a procedure that executes procedure tests corresponding to test-spec.

  can be used as the last expression in a test module

test-lambda

  :: symbol:formals/([arguments expected settings]) body ... -> procedure

  creates a normalised test procedure with the correct arity

assert-and :: [optional-title] expression ... -> vector:test-result

  creates a successful test result if all given expressions or assertions are successful test results or true

assert-equal :: [optional-title] expected expression -> vector:test-result

import name

(sph test)

exports

assert-and

syntax

assert-equal

syntax

signature

optional-title expected expr

assert-test-result

syntax

signature

title expr continue

assert-true

syntax

signature

optional-title expr

define-procedure-tests

syntax

signature

name test-spec ...

define-test

syntax

define-test-module

syntax

sph-test-description

variable

test-create-result

procedure

signature

values ... ->

boolean string integer any list any -> vector

success? title index result arguments expected -> test-result

test-execute-module

procedure

signature

settings name ->

list (symbol ...) -> test-result

test-execute-modules

procedure

signature

settings module-names ->

list list -> test-result

test-execute-modules-by-prefix

procedure

signature

#:settings module-names ... ->

description

execute all test modules whose module name has the one of the given module name prefixes.

"path-search" restricts the path where to search for test-modules.

"search-type" does not execute modules that exactly match the module name prefix (where the module name prefix resolves to a regular file)

test-execute-procedures

procedure

signature

settings source ->

list ((symbol:name procedure:test-proc any:data-in/out ...) ...) -> test-result

test-execute-procedures-lambda

syntax

signature

test-spec ...

test-lambda

syntax

test-list

syntax

signature

test-spec ...

test-module-name-from-files

procedure

signature

a ->

string -> list/error

test-result

variable

test-result-success?

procedure

signature

record ->

test-settings-default

variable

test-settings-default-custom

syntax

signature

key/value ...

test-settings-default-custom-by-list

procedure

signature

key/value ->

[key value] ... -> list

description

get the default test settings, with values possibly set to the values given with "key/value"

test-success?

procedure

signature

result expected ->

vector/any any -> boolean

description

if result is a test-result, check if it is a successful result. otherwise compare result and expected for equality


tags: programming guile documentation library scheme sph-lib q1 test highlight sph-test