package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (40)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-lwt-1.1.0.tbz
sha256=212827a49abf4008581c0da53e7ec78a9d639b415380dcb1fdeeb23f3ff083e2
sha512=c47d04b3c7100af703b470b93ff9fe9fe22790415370b6d5972736f46a5c83901717d67caf0c4115d01020d3078dc7f3063838578174921cab352546dad00148

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 03 Apr 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3" & < "1.1.0"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "2.0"

Dev Dependencies (1)

  1. odoc with-doc

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.0" | = "1.1.0"
  4. alcotest-lwt < "1.0.0" | = "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. ambient-context-lwt
  9. angstrom >= "0.7.0"
  10. ansi >= "0.6.0"
  11. anycache >= "0.7.4"
  12. anycache-async
  13. anycache-lwt
  14. archetype >= "1.4.2"
  15. archi
  16. arp
  17. arp-mirage
  18. arrakis
  19. art
  20. asak >= "0.2"
  21. asli >= "0.2.0"
  22. asn1-combinators >= "0.2.2"
  23. atd >= "2.3.3"
  24. atdgen >= "2.10.0"
  25. atdpy
  26. atdts
  27. base32
  28. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  29. bastet
  30. bastet_async
  31. bastet_lwt
  32. bech32
  33. bechamel >= "0.5.0"
  34. bigarray-overlap
  35. bigstring >= "0.3"
  36. bigstring-unix >= "0.3"
  37. bigstringaf
  38. bitlib
  39. blake2
  40. bloomf
  41. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  42. bls12-381-hash
  43. bls12-381-js >= "0.4.2"
  44. bls12-381-js-gen >= "0.4.2"
  45. bls12-381-legacy
  46. bls12-381-signature
  47. bls12-381-unix
  48. blurhash
  49. builder-web
  50. bulletml
  51. bytebuffer
  52. ca-certs
  53. ca-certs-nss
  54. cactus
  55. caldav
  56. calendar >= "3.0.0"
  57. callipyge
  58. camlix
  59. capnp-rpc < "1.2.3"
  60. capnp-rpc-lwt < "0.3"
  61. capnp-rpc-mirage >= "0.9.0"
  62. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  63. carray
  64. carton
  65. cborl
  66. ccss >= "1.6"
  67. cf-lwt
  68. chacha
  69. channel
  70. charrua-client
  71. charrua-client-lwt
  72. charrua-client-mirage < "0.11.0"
  73. checkseum >= "0.0.3"
  74. cid
  75. clarity-lang
  76. class_group_vdf
  77. cohttp >= "0.17.0"
  78. cohttp-curl-async
  79. cohttp-curl-lwt
  80. cohttp-eio >= "6.0.0~beta2"
  81. colombe >= "0.2.0"
  82. color
  83. conan
  84. conan-cli
  85. conan-database
  86. conan-lwt
  87. conan-unix
  88. conduit = "3.0.0"
  89. conex < "0.10.0"
  90. conex-mirage-crypto
  91. conex-nocrypto
  92. cookie
  93. cow >= "2.2.0"
  94. css
  95. css-parser
  96. cstruct >= "3.3.0"
  97. cstruct-sexp
  98. ctypes-zarith
  99. cuid
  100. curly
  101. current_incr
  102. cwe_checker
  103. data-encoding
  104. datakit >= "0.12.0"
  105. datakit-bridge-github >= "0.12.0"
  106. datakit-ci
  107. datakit-client-git >= "0.12.0"
  108. decompress >= "0.8" & < "1.5.3"
  109. depyt
  110. digestif >= "0.8.1"
  111. dispatch >= "0.4.1"
  112. dkim
  113. dkim-bin
  114. dkim-mirage
  115. dns >= "4.0.0"
  116. dns-cli
  117. dns-client >= "4.6.0"
  118. dns-forward < "0.9.0"
  119. dns-forward-lwt-unix
  120. dns-resolver
  121. dns-server
  122. dns-tsig
  123. dnssd
  124. dnssec
  125. docfd >= "2.2.0"
  126. dog < "0.2.1"
  127. domain-name
  128. dream
  129. dream-pure
  130. duff
  131. dune-release >= "1.0.0"
  132. duration >= "0.1.1"
  133. emile
  134. encore
  135. eqaf >= "0.5"
  136. equinoxe
  137. equinoxe-cohttp
  138. equinoxe-hlc
  139. eris
  140. eris-lwt
  141. ezgzip
  142. ezjsonm >= "0.4.2" & < "1.3.0"
  143. ezjsonm-lwt < "1.3.0"
  144. FPauth
  145. FPauth-core
  146. FPauth-responses
  147. FPauth-strategies
  148. faraday != "0.2.0"
  149. farfadet
  150. fat-filesystem >= "0.12.0"
  151. ff
  152. ff-pbt
  153. fiat-p256
  154. flex-array
  155. fsevents-lwt
  156. functoria >= "2.2.0"
  157. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  158. geojson
  159. geoml >= "0.1.1"
  160. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0"
  161. git-mirage < "3.0.0"
  162. git-unix >= "1.10.0" & != "2.1.0"
  163. gitlab-unix
  164. glicko2
  165. gmap >= "0.3.0"
  166. gobba
  167. gpt
  168. graphql
  169. graphql-async
  170. graphql-cohttp >= "0.13.0"
  171. graphql-lwt
  172. graphql_parser != "0.11.0"
  173. graphql_ppx >= "0.7.1"
  174. h1_parser
  175. h2
  176. hacl
  177. hacl-star >= "0.6.0"
  178. hacl_func
  179. hacl_x25519 >= "0.2.0"
  180. highlexer
  181. hkdf
  182. hockmd
  183. html_of_jsx
  184. http
  185. http-multipart-formdata < "2.0.0"
  186. httpaf >= "0.2.0"
  187. hvsock
  188. icalendar >= "0.1.4"
  189. imagelib >= "20200929"
  190. index
  191. inferno >= "20220603"
  192. influxdb-async
  193. influxdb-lwt
  194. inquire < "0.2.0"
  195. interval-map
  196. iomux
  197. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & != "2.3.0"
  198. irmin-bench >= "2.7.0"
  199. irmin-chunk < "1.3.0" | >= "2.3.0"
  200. irmin-cli
  201. irmin-containers
  202. irmin-fs < "1.3.0" | >= "2.3.0"
  203. irmin-git < "2.0.0" | >= "2.3.0"
  204. irmin-http < "2.0.0"
  205. irmin-mem < "1.3.0"
  206. irmin-pack >= "2.4.0" & != "2.6.1"
  207. irmin-pack-tools
  208. irmin-test >= "2.2.0" & < "3.0.0"
  209. irmin-tezos
  210. irmin-tezos-utils
  211. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  212. irmin-watcher
  213. jekyll-format
  214. jerboa
  215. jitsu
  216. jose
  217. json-data-encoding >= "0.9"
  218. json_decoder
  219. jsonxt
  220. junit_alcotest
  221. jwto
  222. ke >= "0.2"
  223. kkmarkdown
  224. lambda-runtime
  225. lambda_streams
  226. lambda_streams_async
  227. lambdapi >= "2.0.0"
  228. lambdoc >= "1.0-beta4"
  229. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  230. letters
  231. lmdb >= "1.0"
  232. logical
  233. logtk >= "1.6"
  234. lp
  235. lp-glpk
  236. lp-glpk-js
  237. lp-gurobi
  238. lru
  239. lt-code
  240. luv
  241. mbr-format >= "1.0.0"
  242. mdx >= "1.6.0"
  243. mec
  244. mechaml >= "1.0.0"
  245. merge-queues >= "0.2.0"
  246. merge-ropes >= "0.2.0"
  247. metrics
  248. minicaml = "0.3.1" | >= "0.4"
  249. mirage >= "4.0.0~beta1"
  250. mirage-block-partition
  251. mirage-block-ramdisk >= "0.3"
  252. mirage-channel >= "4.0.0"
  253. mirage-channel-lwt
  254. mirage-crypto-ec
  255. mirage-flow >= "1.0.2" & < "1.2.0"
  256. mirage-flow-unix
  257. mirage-fs-mem
  258. mirage-fs-unix >= "1.2.0"
  259. mirage-kv >= "2.0.0"
  260. mirage-kv-mem
  261. mirage-kv-unix
  262. mirage-logs >= "0.3.0"
  263. mirage-nat
  264. mirage-net-unix >= "2.3.0"
  265. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  266. mirage-tc
  267. mjson
  268. mmdb
  269. mnd
  270. monocypher
  271. mrmime >= "0.2.0"
  272. mrt-format
  273. msgpck >= "1.6"
  274. mssql >= "2.0.3"
  275. multibase
  276. multihash
  277. multihash-digestif
  278. multipart-form-data
  279. multipart_form
  280. multipart_form-eio
  281. multipart_form-lwt
  282. named-pipe
  283. nanoid
  284. nbd >= "4.0.3"
  285. nbd-tool
  286. nloge
  287. nocoiner
  288. non_empty_list
  289. OCADml >= "0.6.0"
  290. ocaml-r >= "0.5.0"
  291. ocaml-version >= "3.1.0"
  292. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  293. ocamlformat-rpc < "removed"
  294. ocamline
  295. ocluster < "0.3.0"
  296. odoc >= "1.4.0" & < "2.1.0"
  297. ohex
  298. oidc
  299. opam-0install
  300. opam-file-format >= "2.1.1"
  301. opentelemetry >= "0.6"
  302. opentelemetry-client-cohttp-lwt >= "0.6"
  303. opentelemetry-client-ocurl >= "0.6"
  304. opentelemetry-cohttp-lwt >= "0.6"
  305. opentelemetry-lwt >= "0.6"
  306. opium >= "0.15.0"
  307. opium-graphql
  308. opium-testing
  309. opium_kernel
  310. orewa
  311. ortac-core
  312. osx-acl
  313. osx-attr
  314. osx-cf
  315. osx-fsevents
  316. osx-membership
  317. osx-mount
  318. osx-xattr
  319. otoggl
  320. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  321. owl-base < "0.5.0"
  322. owl-ode >= "0.1.0" & != "0.2.0"
  323. owl-symbolic
  324. passmaker
  325. patch
  326. pbkdf
  327. pecu >= "0.2"
  328. pf-qubes
  329. pg_query >= "0.9.6"
  330. pgx >= "1.0"
  331. pgx_unix >= "1.0"
  332. pgx_value_core
  333. pgx_value_ptime
  334. phylogenetics
  335. piaf
  336. polyglot
  337. polynomial
  338. ppx_blob >= "0.3.0"
  339. ppx_deriving_cmdliner
  340. ppx_deriving_rpc
  341. ppx_deriving_yaml
  342. ppx_graphql >= "0.2.0"
  343. ppx_inline_alcotest
  344. ppx_parser
  345. ppx_protocol_conv >= "5.0.0"
  346. ppx_protocol_conv_json >= "5.0.0"
  347. ppx_protocol_conv_jsonm >= "5.0.0"
  348. ppx_protocol_conv_msgpack >= "5.0.0"
  349. ppx_protocol_conv_xml_light >= "5.0.0"
  350. ppx_protocol_conv_xmlm
  351. ppx_protocol_conv_yaml >= "5.0.0"
  352. ppx_repr < "0.4.0"
  353. ppx_subliner
  354. ppx_units
  355. ppx_yojson >= "1.1.0"
  356. pratter
  357. prc
  358. preface
  359. pretty_expressive
  360. prettym
  361. proc-smaps
  362. producer < "0.2.0"
  363. progress < "0.2.0"
  364. prom
  365. prometheus < "1.2"
  366. prometheus-app
  367. protocell
  368. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  369. protocol-9p-unix
  370. psq
  371. qcheck >= "0.18"
  372. qcheck-alcotest
  373. qcheck-core >= "0.18"
  374. quickjs
  375. radis
  376. randii
  377. reason-standard
  378. reparse >= "2.0.0" & < "3.0.0"
  379. reparse-unix < "2.1.0"
  380. resp
  381. resp-unix >= "0.10.0"
  382. rfc1951 < "1.0.0"
  383. routes < "2.0.0"
  384. rpc >= "7.1.0"
  385. rpclib >= "7.1.0"
  386. rpclib-async
  387. rpclib-lwt >= "7.1.0"
  388. rubytt
  389. SZXX >= "4.0.0"
  390. salsa20
  391. salsa20-core
  392. sanddb >= "0.2"
  393. scaml >= "1.5.0"
  394. scrypt-kdf
  395. secp256k1 >= "0.4.1"
  396. secp256k1-internal
  397. semver >= "0.2.1"
  398. sendmail
  399. sendmail-lwt
  400. sendmsg
  401. server-reason-react
  402. session-cookie
  403. session-cookie-async
  404. session-cookie-lwt
  405. sherlodoc
  406. slug
  407. sodium-fmt
  408. spin >= "0.6.0"
  409. squirrel
  410. ssh-agent
  411. ssl >= "0.6.0"
  412. stramon-lib
  413. syslog-rfc5424
  414. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "8.0.0"
  415. tdigest < "2.1.0"
  416. terminal_size >= "0.1.1"
  417. terminus
  418. terminus-cohttp
  419. terminus-hlc
  420. terml
  421. textrazor
  422. tezos-base-test-helpers < "13.0"
  423. tezos-bls12-381-polynomial
  424. tezos-client-base < "12.0"
  425. tezos-crypto >= "8.0" & < "9.0"
  426. tezos-lmdb
  427. tezos-plompiler = "0.1.3"
  428. tezos-plonk = "0.1.3"
  429. tezos-signer-backends >= "8.0" & < "13.0"
  430. tezos-stdlib >= "8.0" & < "12.0"
  431. tezos-test-helpers < "12.0"
  432. tftp
  433. timedesc
  434. timere
  435. tls >= "0.12.0"
  436. toc
  437. topojson
  438. topojsone
  439. transept
  440. twostep
  441. type_eq
  442. type_id
  443. typebeat
  444. typeid >= "1.0.1"
  445. tyre >= "0.4"
  446. tyxml >= "4.0.0"
  447. tyxml-jsx
  448. tyxml-ppx >= "4.3.0"
  449. tyxml-syntax
  450. uecc
  451. ulid
  452. universal-portal
  453. unix-dirent
  454. unix-errno >= "0.3.0"
  455. unix-fcntl >= "0.3.0"
  456. unix-sys-resource
  457. unix-sys-stat
  458. unix-time
  459. unstrctrd
  460. user-agent-parser
  461. uspf
  462. uspf-lwt
  463. uspf-unix
  464. utop >= "2.13.0"
  465. validate
  466. validator
  467. vercel
  468. vpnkit
  469. websocketaf
  470. x509 >= "0.7.0"
  471. xapi-rrd >= "1.8.2"
  472. xapi-stdext-date
  473. xapi-stdext-encodings
  474. xapi-stdext-std >= "4.16.0"
  475. yaml < "3.2.0"
  476. yaml-sexp
  477. yocaml
  478. yocaml_yaml
  479. yojson >= "1.6.0"
  480. yuscii >= "0.3.0"
  481. zar
  482. zed >= "3.2.2"
  483. zlist < "0.4.0"

Conflicts

None