package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (40)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-1.0.0.tbz
sha256=ddabff1722ddef4a521c89b9572b9d06f2440d89169db806bea848cb218d83a8
sha512=3c9dffbb5064cf3e9995110628c7fdf466651e9e022addc8eb1c79585863112a195c254994eb8f8384e183c9e2d9c946e28dcd4b1cac7ca48114a478de2362c0

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 11 Feb 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "1.11"

Dev Dependencies

None

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.1"
  4. alcotest-lwt < "1.0.1"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. ambient-context-lwt
  9. angstrom >= "0.7.0"
  10. ansi >= "0.6.0"
  11. anycache >= "0.7.4"
  12. anycache-async
  13. anycache-lwt
  14. archetype >= "1.4.2"
  15. archi
  16. arp
  17. arp-mirage
  18. arrakis
  19. art
  20. asak >= "0.2"
  21. asli >= "0.2.0"
  22. asn1-combinators >= "0.2.2"
  23. atd >= "2.3.3"
  24. atdgen >= "2.10.0"
  25. atdpy
  26. atdts
  27. base32
  28. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  29. bechamel >= "0.5.0"
  30. bigarray-overlap
  31. bigstring >= "0.3"
  32. bigstring-unix >= "0.3"
  33. bigstringaf
  34. bitlib
  35. blake2
  36. bloomf
  37. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  38. bls12-381-hash
  39. bls12-381-js >= "0.4.2"
  40. bls12-381-js-gen >= "0.4.2"
  41. bls12-381-legacy
  42. bls12-381-signature
  43. bls12-381-unix
  44. blurhash
  45. builder-web
  46. bulletml
  47. bytebuffer
  48. ca-certs
  49. ca-certs-nss
  50. cactus
  51. caldav
  52. calendar >= "3.0.0"
  53. callipyge
  54. camlix
  55. capnp-rpc < "0.6.0"
  56. capnp-rpc-lwt < "0.3"
  57. carray
  58. carton
  59. cborl
  60. ccss >= "1.6"
  61. cf-lwt
  62. chacha
  63. channel
  64. charrua-client
  65. charrua-client-lwt
  66. charrua-client-mirage < "0.11.0"
  67. checked_oint
  68. checkseum >= "0.0.3"
  69. cid
  70. clarity-lang
  71. class_group_vdf
  72. cohttp >= "0.17.0"
  73. cohttp-curl-async
  74. cohttp-curl-lwt
  75. cohttp-eio >= "6.0.0~beta2"
  76. colombe >= "0.2.0"
  77. color
  78. conan
  79. conan-cli
  80. conan-database
  81. conan-lwt
  82. conan-unix
  83. conduit = "3.0.0"
  84. conex < "0.10.0"
  85. conex-mirage-crypto
  86. conex-nocrypto
  87. cookie
  88. cow >= "2.2.0"
  89. css
  90. css-parser
  91. cstruct >= "3.3.0"
  92. cstruct-sexp
  93. ctypes-zarith
  94. cuid
  95. curly
  96. current_incr
  97. cwe_checker
  98. data-encoding
  99. datakit >= "0.12.0"
  100. datakit-bridge-github >= "0.12.0"
  101. datakit-ci
  102. datakit-client-git >= "0.12.0"
  103. decompress >= "0.8" & < "1.5.3"
  104. depyt
  105. digestif >= "0.8.1"
  106. dispatch >= "0.4.1"
  107. dkim
  108. dkim-bin
  109. dkim-mirage
  110. dns >= "4.0.0"
  111. dns-cli
  112. dns-client >= "4.6.0"
  113. dns-forward < "0.9.0"
  114. dns-forward-lwt-unix
  115. dns-resolver
  116. dns-server
  117. dns-tsig
  118. dnssd
  119. dnssec
  120. docfd >= "2.2.0"
  121. dog < "0.2.1"
  122. domain-name
  123. dream
  124. dream-pure
  125. duff
  126. dune-release >= "1.0.0"
  127. duration >= "0.1.1"
  128. emile
  129. encore
  130. eqaf >= "0.5"
  131. equinoxe
  132. equinoxe-cohttp
  133. equinoxe-hlc
  134. eris
  135. eris-lwt
  136. ezgzip
  137. ezjsonm >= "0.4.2" & < "1.3.0"
  138. ezjsonm-lwt < "1.3.0"
  139. FPauth
  140. FPauth-core
  141. FPauth-responses
  142. FPauth-strategies
  143. faraday != "0.2.0"
  144. farfadet
  145. fat-filesystem >= "0.12.0"
  146. ff
  147. ff-pbt
  148. fiat-p256
  149. flex-array
  150. fsevents-lwt
  151. functoria >= "2.2.0"
  152. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  153. geojson
  154. geoml >= "0.1.1"
  155. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0" & < "3.0.0"
  156. git-mirage < "3.0.0"
  157. git-unix >= "1.10.0" & != "2.1.0" & < "3.0.0"
  158. gitlab-unix
  159. glicko2
  160. gmap >= "0.3.0"
  161. gobba
  162. gpt
  163. graphql
  164. graphql-async
  165. graphql-cohttp >= "0.13.0"
  166. graphql-lwt
  167. graphql_parser != "0.11.0"
  168. graphql_ppx >= "0.7.1"
  169. h1_parser
  170. h2
  171. hacl
  172. hacl_func
  173. hacl_x25519 >= "0.2.0"
  174. highlexer
  175. hkdf
  176. hockmd
  177. html_of_jsx
  178. http
  179. http-multipart-formdata < "2.0.0"
  180. httpaf >= "0.2.0"
  181. hvsock
  182. icalendar >= "0.1.4"
  183. imagelib >= "20200929"
  184. index
  185. inferno >= "20220603"
  186. influxdb-async
  187. influxdb-lwt
  188. inquire < "0.2.0"
  189. interval-map
  190. iomux
  191. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & < "2.3.0"
  192. irmin-bench >= "2.7.0"
  193. irmin-chunk < "1.3.0" | >= "2.3.0"
  194. irmin-cli
  195. irmin-containers
  196. irmin-fs < "1.3.0" | >= "2.3.0"
  197. irmin-git < "2.0.0" | >= "2.3.0"
  198. irmin-http < "2.0.0"
  199. irmin-mem < "1.3.0"
  200. irmin-pack >= "2.4.0" & != "2.6.1"
  201. irmin-pack-tools
  202. irmin-tezos
  203. irmin-tezos-utils
  204. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  205. irmin-watcher
  206. jekyll-format
  207. jerboa
  208. jitsu
  209. jose
  210. json-data-encoding >= "0.9"
  211. json_decoder
  212. jsonxt
  213. junit_alcotest
  214. jwto
  215. ke >= "0.2"
  216. kkmarkdown
  217. lambda-runtime
  218. lambdapi >= "2.0.0"
  219. lambdoc >= "1.0-beta4"
  220. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  221. lmdb >= "1.0"
  222. logical
  223. logtk >= "1.6"
  224. lp
  225. lp-glpk
  226. lp-glpk-js
  227. lp-gurobi
  228. lru
  229. lt-code
  230. luv
  231. mbr-format >= "1.0.0"
  232. mdx >= "1.6.0"
  233. mec
  234. mechaml >= "1.0.0"
  235. merge-queues >= "0.2.0"
  236. merge-ropes >= "0.2.0"
  237. metrics
  238. minicaml = "0.3.1" | >= "0.4"
  239. mirage >= "4.0.0~beta1"
  240. mirage-block-partition
  241. mirage-block-ramdisk >= "0.3"
  242. mirage-channel >= "4.0.0"
  243. mirage-channel-lwt
  244. mirage-crypto-ec
  245. mirage-flow >= "1.0.2" & < "1.2.0"
  246. mirage-flow-unix
  247. mirage-fs-mem
  248. mirage-fs-unix >= "1.2.0"
  249. mirage-kv >= "2.0.0"
  250. mirage-kv-mem
  251. mirage-kv-unix
  252. mirage-logs >= "0.3.0"
  253. mirage-nat
  254. mirage-net-unix >= "2.3.0"
  255. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  256. mirage-tc
  257. mjson
  258. mmdb
  259. mnd
  260. monocypher
  261. mrmime >= "0.2.0"
  262. mrt-format
  263. msgpck >= "1.6"
  264. multibase
  265. multihash
  266. multihash-digestif
  267. multipart-form-data
  268. multipart_form
  269. multipart_form-eio
  270. multipart_form-lwt
  271. named-pipe
  272. nanoid
  273. nbd >= "4.0.3"
  274. nbd-tool
  275. nloge
  276. nocoiner
  277. non_empty_list
  278. OCADml >= "0.6.0"
  279. ocaml-r >= "0.5.0"
  280. ocaml-version >= "3.1.0"
  281. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  282. ocamlformat-rpc < "removed"
  283. ocamline
  284. ocluster < "0.3.0"
  285. odoc >= "1.4.0" & < "2.1.0"
  286. ohex
  287. oidc
  288. opam-0install
  289. opam-file-format >= "2.1.1"
  290. opentelemetry >= "0.6"
  291. opentelemetry-client-cohttp-lwt >= "0.6"
  292. opentelemetry-client-ocurl >= "0.6"
  293. opentelemetry-cohttp-lwt >= "0.6"
  294. opentelemetry-lwt >= "0.6"
  295. opium >= "0.15.0"
  296. opium-graphql
  297. opium-testing
  298. opium_kernel
  299. orewa
  300. ortac-core
  301. osx-acl
  302. osx-attr
  303. osx-cf
  304. osx-fsevents
  305. osx-membership
  306. osx-mount
  307. osx-xattr
  308. otoggl
  309. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  310. owl-base < "0.5.0"
  311. owl-ode >= "0.1.0" & != "0.2.0"
  312. owl-symbolic
  313. passmaker
  314. patch
  315. pbkdf
  316. pecu >= "0.2"
  317. pf-qubes
  318. pg_query >= "0.9.6"
  319. pgx >= "1.0"
  320. pgx_unix >= "1.0"
  321. pgx_value_core
  322. pgx_value_ptime
  323. phylogenetics
  324. piaf
  325. polyglot
  326. polynomial
  327. ppx_blob >= "0.3.0"
  328. ppx_deriving_cmdliner
  329. ppx_deriving_rpc
  330. ppx_deriving_yaml
  331. ppx_graphql >= "0.2.0"
  332. ppx_inline_alcotest
  333. ppx_parser
  334. ppx_protocol_conv >= "5.0.0"
  335. ppx_protocol_conv_json >= "5.0.0"
  336. ppx_protocol_conv_jsonm >= "5.0.0"
  337. ppx_protocol_conv_msgpack >= "5.0.0"
  338. ppx_protocol_conv_xml_light >= "5.0.0"
  339. ppx_protocol_conv_xmlm
  340. ppx_protocol_conv_yaml >= "5.0.0"
  341. ppx_subliner
  342. ppx_units
  343. ppx_yojson >= "1.1.0"
  344. pratter
  345. prc
  346. preface
  347. pretty_expressive
  348. prettym
  349. proc-smaps
  350. producer < "0.2.0"
  351. prom < "0.2"
  352. prometheus < "1.2"
  353. prometheus-app
  354. protocell
  355. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  356. protocol-9p-unix
  357. psq
  358. qcheck >= "0.18"
  359. qcheck-alcotest
  360. qcheck-core >= "0.18"
  361. quickjs
  362. radis
  363. randii
  364. reason-standard
  365. reparse >= "2.0.0" & < "3.0.0"
  366. reparse-unix < "2.1.0"
  367. resp < "0.10.0"
  368. resp-unix >= "0.10.0"
  369. rfc1951 < "1.0.0"
  370. routes < "2.0.0"
  371. rpc >= "7.1.0"
  372. rpclib >= "7.1.0"
  373. rpclib-async
  374. rpclib-lwt >= "7.1.0"
  375. rubytt
  376. SZXX >= "4.0.0"
  377. salsa20
  378. salsa20-core
  379. sanddb >= "0.2"
  380. scaml >= "1.5.0"
  381. scrypt-kdf
  382. secp256k1 >= "0.4.1"
  383. secp256k1-internal
  384. semver >= "0.2.1"
  385. sendmail
  386. sendmail-lwt
  387. sendmsg
  388. server-reason-react
  389. session-cookie
  390. session-cookie-async
  391. session-cookie-lwt
  392. sherlodoc
  393. slug
  394. sodium-fmt
  395. spin >= "0.6.0"
  396. squirrel
  397. ssh-agent
  398. ssl >= "0.6.0"
  399. stramon-lib
  400. styled-ppx
  401. syslog-rfc5424
  402. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "8.0.0"
  403. tdigest < "2.1.0"
  404. terminal_size >= "0.1.1"
  405. terminus
  406. terminus-cohttp
  407. terminus-hlc
  408. terml
  409. textrazor
  410. tezos-base-test-helpers < "13.0"
  411. tezos-client-base < "12.0"
  412. tezos-lmdb
  413. tezos-test-helpers < "11.0"
  414. tftp
  415. timedesc
  416. timere
  417. tls >= "0.12.0"
  418. toc
  419. topojson
  420. topojsone
  421. transept
  422. twostep
  423. type_eq
  424. type_id
  425. typebeat
  426. typeid >= "1.0.1"
  427. tyre >= "0.4"
  428. tyxml >= "4.0.0"
  429. tyxml-jsx
  430. tyxml-ppx >= "4.3.0"
  431. tyxml-syntax
  432. uecc
  433. ulid
  434. universal-portal
  435. unix-dirent
  436. unix-errno >= "0.3.0"
  437. unix-fcntl >= "0.3.0"
  438. unix-sys-resource
  439. unix-sys-stat
  440. unix-time
  441. unstrctrd
  442. user-agent-parser
  443. uspf
  444. uspf-lwt
  445. uspf-unix
  446. utop >= "2.13.0"
  447. validate
  448. validator
  449. vercel
  450. vpnkit
  451. wcwidth
  452. websocketaf
  453. x509 >= "0.7.0"
  454. xapi-rrd >= "1.8.2"
  455. xapi-stdext-date
  456. xapi-stdext-encodings
  457. xapi-stdext-std >= "4.16.0"
  458. yaml < "3.2.0"
  459. yaml-sexp
  460. yocaml
  461. yocaml_yaml
  462. yojson >= "1.6.0"
  463. yuscii >= "0.3.0"
  464. zar
  465. zed >= "3.2.2"
  466. zlist < "0.4.0"

Conflicts

None