package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (40)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-1.0.1.tbz
sha256=0c8748838a89df6dee4850aa7ef5e46c573265a9bf1589dec255bd8156a793f6
sha512=f5f52dea5bb143e7001b8d0ac6131f8851389b080f46b9ad1ccacb95cc31a38143dd7122ccba59bb190abe559dbf81f33cc4dc3401ed95772d59be75fa566f19

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 13 Feb 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "1.11"

Dev Dependencies

None

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async != "1.0.0" & < "1.1.0"
  4. alcotest-lwt != "1.0.0" & < "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. ambient-context-lwt
  9. angstrom >= "0.7.0"
  10. ansi >= "0.6.0"
  11. anycache >= "0.7.4"
  12. anycache-async
  13. anycache-lwt
  14. archetype >= "1.4.2"
  15. archi
  16. arp
  17. arp-mirage
  18. arrakis
  19. art
  20. asak >= "0.2"
  21. asli >= "0.2.0"
  22. asn1-combinators >= "0.2.2"
  23. atd >= "2.3.3"
  24. atdgen >= "2.10.0"
  25. atdpy
  26. atdts
  27. base32
  28. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  29. bastet
  30. bastet_async
  31. bastet_lwt
  32. bechamel >= "0.5.0"
  33. bigarray-overlap
  34. bigstring >= "0.3"
  35. bigstring-unix >= "0.3"
  36. bigstringaf
  37. bitlib
  38. blake2
  39. bloomf
  40. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  41. bls12-381-hash
  42. bls12-381-js >= "0.4.2"
  43. bls12-381-js-gen >= "0.4.2"
  44. bls12-381-legacy
  45. bls12-381-signature
  46. bls12-381-unix
  47. blurhash
  48. builder-web
  49. bulletml
  50. bytebuffer
  51. ca-certs
  52. ca-certs-nss
  53. cactus
  54. caldav
  55. calendar >= "3.0.0"
  56. callipyge
  57. camlix
  58. capnp-rpc < "1.2.3"
  59. capnp-rpc-lwt < "0.3"
  60. capnp-rpc-mirage >= "0.9.0"
  61. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  62. carray
  63. carton
  64. cborl
  65. ccss >= "1.6"
  66. cf-lwt
  67. chacha
  68. channel
  69. charrua-client
  70. charrua-client-lwt
  71. charrua-client-mirage < "0.11.0"
  72. checked_oint
  73. checkseum >= "0.0.3"
  74. cid
  75. clarity-lang
  76. class_group_vdf
  77. cohttp >= "0.17.0"
  78. cohttp-curl-async
  79. cohttp-curl-lwt
  80. cohttp-eio >= "6.0.0~beta2"
  81. colombe >= "0.2.0"
  82. color
  83. conan
  84. conan-cli
  85. conan-database
  86. conan-lwt
  87. conan-unix
  88. conduit = "3.0.0"
  89. conex < "0.10.0"
  90. conex-mirage-crypto
  91. conex-nocrypto
  92. cookie
  93. cow >= "2.2.0"
  94. css
  95. css-parser
  96. cstruct >= "3.3.0"
  97. cstruct-sexp
  98. ctypes-zarith
  99. cuid
  100. curly
  101. current_incr
  102. cwe_checker
  103. data-encoding
  104. datakit >= "0.12.0"
  105. datakit-bridge-github >= "0.12.0"
  106. datakit-ci
  107. datakit-client-git >= "0.12.0"
  108. decompress >= "0.8" & < "1.5.3"
  109. depyt
  110. digestif >= "0.8.1"
  111. dispatch >= "0.4.1"
  112. dkim
  113. dkim-bin
  114. dkim-mirage
  115. dns >= "4.0.0"
  116. dns-cli
  117. dns-client >= "4.6.0"
  118. dns-forward < "0.9.0"
  119. dns-forward-lwt-unix
  120. dns-resolver
  121. dns-server
  122. dns-tsig
  123. dnssd
  124. dnssec
  125. docfd >= "2.2.0"
  126. dog < "0.2.1"
  127. domain-name
  128. dream
  129. dream-pure
  130. duff
  131. dune-release >= "1.0.0"
  132. duration >= "0.1.1"
  133. emile
  134. encore
  135. eqaf >= "0.5"
  136. equinoxe
  137. equinoxe-cohttp
  138. equinoxe-hlc
  139. eris
  140. eris-lwt
  141. ezgzip
  142. ezjsonm >= "0.4.2" & < "1.3.0"
  143. ezjsonm-lwt < "1.3.0"
  144. FPauth
  145. FPauth-core
  146. FPauth-responses
  147. FPauth-strategies
  148. faraday != "0.2.0"
  149. farfadet
  150. fat-filesystem >= "0.12.0"
  151. ff
  152. ff-pbt
  153. fiat-p256
  154. flex-array
  155. fsevents-lwt
  156. functoria >= "2.2.0"
  157. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  158. geojson
  159. geoml >= "0.1.1"
  160. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0" & < "3.0.0"
  161. git-mirage < "3.0.0"
  162. git-unix >= "1.10.0" & != "2.1.0" & < "3.0.0"
  163. gitlab-unix
  164. glicko2
  165. gmap >= "0.3.0"
  166. gobba
  167. gpt
  168. graphql
  169. graphql-async
  170. graphql-cohttp >= "0.13.0"
  171. graphql-lwt
  172. graphql_parser != "0.11.0"
  173. graphql_ppx >= "0.7.1"
  174. h1_parser
  175. h2
  176. hacl
  177. hacl_func
  178. hacl_x25519 >= "0.2.0"
  179. highlexer
  180. hkdf
  181. hockmd
  182. html_of_jsx
  183. http
  184. http-multipart-formdata < "2.0.0"
  185. httpaf >= "0.2.0"
  186. hvsock
  187. icalendar >= "0.1.4"
  188. imagelib >= "20200929"
  189. index
  190. inferno >= "20220603"
  191. influxdb-async
  192. influxdb-lwt
  193. inquire < "0.2.0"
  194. interval-map
  195. iomux
  196. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & < "2.3.0"
  197. irmin-bench >= "2.7.0"
  198. irmin-chunk < "1.3.0" | >= "2.3.0"
  199. irmin-cli
  200. irmin-containers
  201. irmin-fs < "1.3.0" | >= "2.3.0"
  202. irmin-git < "2.0.0" | >= "2.3.0"
  203. irmin-http < "2.0.0"
  204. irmin-mem < "1.3.0"
  205. irmin-pack >= "2.4.0" & != "2.6.1"
  206. irmin-pack-tools
  207. irmin-test >= "2.2.0" & < "3.0.0"
  208. irmin-tezos
  209. irmin-tezos-utils
  210. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  211. irmin-watcher
  212. jekyll-format
  213. jerboa
  214. jitsu
  215. jose
  216. json-data-encoding >= "0.9"
  217. json_decoder
  218. jsonxt
  219. junit_alcotest
  220. jwto
  221. ke >= "0.2"
  222. kkmarkdown
  223. lambda-runtime
  224. lambda_streams
  225. lambda_streams_async
  226. lambda_streams_lwt
  227. lambdapi >= "2.0.0"
  228. lambdoc >= "1.0-beta4"
  229. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  230. lmdb >= "1.0"
  231. logical
  232. logtk >= "1.6"
  233. lp
  234. lp-glpk
  235. lp-glpk-js
  236. lp-gurobi
  237. lru
  238. lt-code
  239. luv
  240. mbr-format >= "1.0.0"
  241. mdx >= "1.6.0"
  242. mec
  243. mechaml >= "1.0.0"
  244. merge-queues >= "0.2.0"
  245. merge-ropes >= "0.2.0"
  246. metrics
  247. minicaml = "0.3.1" | >= "0.4"
  248. mirage >= "4.0.0~beta1"
  249. mirage-block-partition
  250. mirage-block-ramdisk >= "0.3"
  251. mirage-channel >= "4.0.0"
  252. mirage-channel-lwt
  253. mirage-crypto-ec
  254. mirage-flow >= "1.0.2" & < "1.2.0"
  255. mirage-flow-unix
  256. mirage-fs-mem
  257. mirage-fs-unix >= "1.2.0"
  258. mirage-kv >= "2.0.0"
  259. mirage-kv-mem
  260. mirage-kv-unix
  261. mirage-logs >= "0.3.0"
  262. mirage-nat
  263. mirage-net-unix >= "2.3.0"
  264. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  265. mirage-tc
  266. mjson
  267. mmdb
  268. mnd
  269. monocypher
  270. mrmime >= "0.2.0"
  271. mrt-format
  272. msgpck >= "1.6"
  273. mssql >= "2.0.3"
  274. multibase
  275. multihash
  276. multihash-digestif
  277. multipart-form-data
  278. multipart_form
  279. multipart_form-eio
  280. multipart_form-lwt
  281. named-pipe
  282. nanoid
  283. nbd >= "4.0.3"
  284. nbd-tool
  285. nloge
  286. nocoiner
  287. non_empty_list
  288. OCADml >= "0.6.0"
  289. ocaml-r >= "0.5.0"
  290. ocaml-version >= "3.1.0"
  291. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  292. ocamlformat-rpc < "removed"
  293. ocamline
  294. ocluster < "0.3.0"
  295. odoc >= "1.4.0" & < "2.1.0"
  296. ohex
  297. oidc
  298. opam-0install
  299. opam-file-format >= "2.1.1"
  300. opentelemetry >= "0.6"
  301. opentelemetry-client-cohttp-lwt >= "0.6"
  302. opentelemetry-client-ocurl >= "0.6"
  303. opentelemetry-cohttp-lwt >= "0.6"
  304. opentelemetry-lwt >= "0.6"
  305. opium >= "0.15.0"
  306. opium-graphql
  307. opium-testing
  308. opium_kernel
  309. orewa
  310. ortac-core
  311. osx-acl
  312. osx-attr
  313. osx-cf
  314. osx-fsevents
  315. osx-membership
  316. osx-mount
  317. osx-xattr
  318. otoggl
  319. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  320. owl-base < "0.5.0"
  321. owl-ode >= "0.1.0" & != "0.2.0"
  322. owl-symbolic
  323. passmaker
  324. patch
  325. pbkdf
  326. pecu >= "0.2"
  327. pf-qubes
  328. pg_query >= "0.9.6"
  329. pgx >= "1.0"
  330. pgx_unix >= "1.0"
  331. pgx_value_core
  332. pgx_value_ptime
  333. phylogenetics
  334. piaf
  335. polyglot
  336. polynomial
  337. ppx_blob >= "0.3.0"
  338. ppx_deriving_cmdliner
  339. ppx_deriving_rpc
  340. ppx_deriving_yaml
  341. ppx_graphql >= "0.2.0"
  342. ppx_inline_alcotest
  343. ppx_parser
  344. ppx_protocol_conv >= "5.0.0"
  345. ppx_protocol_conv_json >= "5.0.0"
  346. ppx_protocol_conv_jsonm >= "5.0.0"
  347. ppx_protocol_conv_msgpack >= "5.0.0"
  348. ppx_protocol_conv_xml_light >= "5.0.0"
  349. ppx_protocol_conv_xmlm
  350. ppx_protocol_conv_yaml >= "5.0.0"
  351. ppx_subliner
  352. ppx_units
  353. ppx_yojson >= "1.1.0"
  354. pratter
  355. prc
  356. preface
  357. pretty_expressive
  358. prettym
  359. proc-smaps
  360. producer < "0.2.0"
  361. prom
  362. prometheus < "1.2"
  363. prometheus-app
  364. protocell
  365. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  366. protocol-9p-unix
  367. psq
  368. qcheck >= "0.18"
  369. qcheck-alcotest
  370. qcheck-core >= "0.18"
  371. quickjs
  372. radis
  373. randii
  374. reason-standard
  375. reparse >= "2.0.0" & < "3.0.0"
  376. reparse-unix < "2.1.0"
  377. resp
  378. resp-unix >= "0.10.0"
  379. rfc1951 < "1.0.0"
  380. routes < "2.0.0"
  381. rpc >= "7.1.0"
  382. rpclib >= "7.1.0"
  383. rpclib-async
  384. rpclib-lwt >= "7.1.0"
  385. rubytt
  386. SZXX >= "4.0.0"
  387. salsa20
  388. salsa20-core
  389. sanddb >= "0.2"
  390. scaml >= "1.5.0"
  391. scrypt-kdf
  392. secp256k1 >= "0.4.1"
  393. secp256k1-internal
  394. semver >= "0.2.1"
  395. sendmail
  396. sendmail-lwt
  397. sendmsg
  398. server-reason-react
  399. session-cookie
  400. session-cookie-async
  401. session-cookie-lwt
  402. sherlodoc
  403. slug
  404. sodium-fmt
  405. spin >= "0.6.0"
  406. squirrel
  407. ssh-agent
  408. ssl >= "0.6.0"
  409. stramon-lib
  410. styled-ppx
  411. syslog-rfc5424
  412. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "8.0.0"
  413. tdigest < "2.1.0"
  414. terminal_size >= "0.1.1"
  415. terminus
  416. terminus-cohttp
  417. terminus-hlc
  418. terml
  419. textrazor
  420. tezos-base-test-helpers < "13.0"
  421. tezos-client-base < "12.0"
  422. tezos-lmdb
  423. tezos-test-helpers < "11.0"
  424. tftp
  425. timedesc
  426. timere
  427. tls >= "0.12.0"
  428. toc
  429. topojson
  430. topojsone
  431. transept
  432. twostep
  433. type_eq
  434. type_id
  435. typebeat
  436. typeid >= "1.0.1"
  437. tyre >= "0.4"
  438. tyxml >= "4.0.0"
  439. tyxml-jsx
  440. tyxml-ppx >= "4.3.0"
  441. tyxml-syntax
  442. uecc
  443. ulid
  444. universal-portal
  445. unix-dirent
  446. unix-errno >= "0.3.0"
  447. unix-fcntl >= "0.3.0"
  448. unix-sys-resource
  449. unix-sys-stat
  450. unix-time
  451. unstrctrd
  452. user-agent-parser
  453. uspf
  454. uspf-lwt
  455. uspf-unix
  456. utop >= "2.13.0"
  457. validate
  458. validator
  459. vercel
  460. vpnkit
  461. wcwidth
  462. websocketaf
  463. x509 >= "0.7.0"
  464. xapi-rrd >= "1.8.2"
  465. xapi-stdext-date
  466. xapi-stdext-encodings
  467. xapi-stdext-std >= "4.16.0"
  468. yaml < "3.2.0"
  469. yaml-sexp
  470. yocaml
  471. yocaml_yaml
  472. yojson >= "1.6.0"
  473. yuscii >= "0.3.0"
  474. zar
  475. zed >= "3.2.2"
  476. zlist < "0.4.0"

Conflicts

None