package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (40)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-lwt-1.1.0.tbz
sha256=212827a49abf4008581c0da53e7ec78a9d639b415380dcb1fdeeb23f3ff083e2
sha512=c47d04b3c7100af703b470b93ff9fe9fe22790415370b6d5972736f46a5c83901717d67caf0c4115d01020d3078dc7f3063838578174921cab352546dad00148

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 03 Apr 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3" & < "1.1.0"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "2.0"

Dev Dependencies (1)

  1. odoc with-doc

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.0" | = "1.1.0"
  4. alcotest-lwt < "1.0.0" | = "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. ambient-context-lwt
  9. angstrom >= "0.7.0"
  10. ansi >= "0.6.0"
  11. anycache >= "0.7.4"
  12. anycache-async
  13. anycache-lwt
  14. archetype >= "1.4.2"
  15. archi
  16. arp
  17. arp-mirage
  18. arrakis
  19. art
  20. asak >= "0.2"
  21. asli >= "0.2.0"
  22. asn1-combinators >= "0.2.2"
  23. atd >= "2.3.3"
  24. atdgen >= "2.10.0"
  25. atdpy
  26. atdts
  27. base32
  28. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  29. bastet
  30. bastet_async
  31. bastet_lwt
  32. bech32
  33. bechamel >= "0.5.0"
  34. bigarray-overlap
  35. bigstring >= "0.3"
  36. bigstring-unix >= "0.3"
  37. bigstringaf
  38. bitlib
  39. blake2
  40. bloomf
  41. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  42. bls12-381-hash
  43. bls12-381-js >= "0.4.2"
  44. bls12-381-js-gen >= "0.4.2"
  45. bls12-381-legacy
  46. bls12-381-signature
  47. bls12-381-unix
  48. blurhash
  49. builder-web
  50. bulletml
  51. bytebuffer
  52. ca-certs
  53. ca-certs-nss
  54. cactus
  55. caldav
  56. calendar >= "3.0.0"
  57. callipyge
  58. camlix
  59. camlkit
  60. camlkit-base
  61. capnp-rpc < "1.2.3"
  62. capnp-rpc-lwt < "0.3"
  63. capnp-rpc-mirage >= "0.9.0"
  64. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  65. carray
  66. carton
  67. cborl
  68. ccss >= "1.6"
  69. cf-lwt
  70. chacha
  71. channel
  72. charrua-client
  73. charrua-client-lwt
  74. charrua-client-mirage < "0.11.0"
  75. checked_oint < "0.1.1"
  76. checkseum >= "0.0.3"
  77. cid
  78. clarity-lang
  79. class_group_vdf
  80. cohttp >= "0.17.0"
  81. cohttp-curl-async
  82. cohttp-curl-lwt
  83. cohttp-eio >= "6.0.0~beta2"
  84. colombe >= "0.2.0"
  85. color
  86. conan
  87. conan-cli
  88. conan-database
  89. conan-lwt
  90. conan-unix
  91. conduit = "3.0.0"
  92. conex < "0.10.0"
  93. conex-mirage-crypto
  94. conex-nocrypto
  95. cookie
  96. cow >= "2.2.0"
  97. css
  98. css-parser
  99. cstruct >= "3.3.0"
  100. cstruct-sexp
  101. ctypes-zarith
  102. cuid
  103. curly
  104. current_incr
  105. cwe_checker
  106. data-encoding
  107. datakit >= "0.12.0"
  108. datakit-bridge-github >= "0.12.0"
  109. datakit-ci
  110. datakit-client-git >= "0.12.0"
  111. decompress >= "0.8" & < "1.5.3"
  112. depyt
  113. digestif >= "0.8.1"
  114. dispatch >= "0.4.1"
  115. dkim
  116. dkim-bin
  117. dkim-mirage
  118. dns >= "4.0.0"
  119. dns-cli
  120. dns-client >= "4.6.0"
  121. dns-forward < "0.9.0"
  122. dns-forward-lwt-unix
  123. dns-resolver
  124. dns-server
  125. dns-tsig
  126. dnssd
  127. dnssec
  128. docfd >= "2.2.0"
  129. dog < "0.2.1"
  130. domain-name
  131. dream
  132. dream-pure
  133. duff
  134. dune-release >= "1.0.0"
  135. duration >= "0.1.1"
  136. emile
  137. encore
  138. eqaf >= "0.5"
  139. equinoxe
  140. equinoxe-cohttp
  141. equinoxe-hlc
  142. eris
  143. eris-lwt
  144. ezgzip
  145. ezjsonm >= "0.4.2" & < "1.3.0"
  146. ezjsonm-lwt < "1.3.0"
  147. FPauth
  148. FPauth-core
  149. FPauth-responses
  150. FPauth-strategies
  151. faraday != "0.2.0"
  152. farfadet
  153. fat-filesystem >= "0.12.0"
  154. ff
  155. ff-pbt
  156. fiat-p256
  157. flex-array
  158. fsevents-lwt
  159. functoria >= "2.2.0"
  160. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  161. geojson
  162. geoml >= "0.1.1"
  163. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0"
  164. git-mirage < "3.0.0"
  165. git-unix >= "1.10.0" & != "2.1.0"
  166. gitlab-unix
  167. glicko2
  168. gmap >= "0.3.0"
  169. gobba
  170. gpt
  171. graphql
  172. graphql-async
  173. graphql-cohttp >= "0.13.0"
  174. graphql-lwt
  175. graphql_parser != "0.11.0"
  176. graphql_ppx >= "0.7.1"
  177. h1_parser
  178. h2
  179. hacl
  180. hacl-star >= "0.6.0"
  181. hacl_func
  182. hacl_x25519 >= "0.2.0"
  183. highlexer
  184. hkdf
  185. hockmd
  186. html_of_jsx
  187. http
  188. http-multipart-formdata < "2.0.0"
  189. httpaf >= "0.2.0"
  190. hvsock
  191. icalendar >= "0.1.4"
  192. imagelib >= "20200929"
  193. index
  194. inferno >= "20220603"
  195. influxdb-async
  196. influxdb-lwt
  197. inquire < "0.2.0"
  198. interval-map
  199. iomux
  200. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & != "2.3.0"
  201. irmin-bench >= "2.7.0"
  202. irmin-chunk < "1.3.0" | >= "2.3.0"
  203. irmin-cli
  204. irmin-containers
  205. irmin-fs < "1.3.0" | >= "2.3.0"
  206. irmin-git < "2.0.0" | >= "2.3.0"
  207. irmin-http < "2.0.0"
  208. irmin-mem < "1.3.0"
  209. irmin-pack >= "2.4.0" & != "2.6.1"
  210. irmin-pack-tools
  211. irmin-test >= "2.2.0" & < "3.0.0"
  212. irmin-tezos
  213. irmin-tezos-utils
  214. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  215. irmin-watcher
  216. jekyll-format
  217. jerboa
  218. jitsu
  219. jose
  220. json-data-encoding >= "0.9"
  221. json_decoder
  222. jsonxt
  223. junit_alcotest
  224. jwto
  225. ke >= "0.2"
  226. kkmarkdown
  227. lambda-runtime
  228. lambda_streams
  229. lambda_streams_async
  230. lambdapi >= "2.0.0"
  231. lambdoc >= "1.0-beta4"
  232. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  233. letters
  234. lmdb >= "1.0"
  235. logical
  236. logtk >= "1.6"
  237. lp
  238. lp-glpk
  239. lp-glpk-js
  240. lp-gurobi
  241. lru
  242. lt-code
  243. luv
  244. mbr-format >= "1.0.0"
  245. mdx >= "1.6.0"
  246. mec
  247. mechaml >= "1.0.0"
  248. merge-queues >= "0.2.0"
  249. merge-ropes >= "0.2.0"
  250. metrics
  251. minicaml = "0.3.1" | >= "0.4"
  252. mirage >= "4.0.0~beta1"
  253. mirage-block-partition
  254. mirage-block-ramdisk >= "0.3"
  255. mirage-channel >= "4.0.0"
  256. mirage-channel-lwt
  257. mirage-crypto-ec
  258. mirage-flow >= "1.0.2" & < "1.2.0"
  259. mirage-flow-unix
  260. mirage-fs-mem
  261. mirage-fs-unix >= "1.2.0"
  262. mirage-kv >= "2.0.0"
  263. mirage-kv-mem
  264. mirage-kv-unix
  265. mirage-logs >= "0.3.0"
  266. mirage-nat
  267. mirage-net-unix >= "2.3.0"
  268. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  269. mirage-tc
  270. mjson
  271. mmdb
  272. mnd
  273. monocypher
  274. mrmime >= "0.2.0"
  275. mrt-format
  276. msgpck >= "1.6"
  277. mssql >= "2.0.3"
  278. multibase
  279. multihash
  280. multihash-digestif
  281. multipart-form-data
  282. multipart_form
  283. multipart_form-eio
  284. multipart_form-lwt
  285. named-pipe
  286. nanoid
  287. nbd >= "4.0.3"
  288. nbd-tool
  289. nloge
  290. nocoiner
  291. non_empty_list
  292. OCADml >= "0.6.0"
  293. ocaml-r >= "0.5.0"
  294. ocaml-version >= "3.1.0"
  295. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  296. ocamlformat-rpc < "removed"
  297. ocamline
  298. ocluster < "0.3.0"
  299. odoc >= "1.4.0" & < "2.1.0"
  300. ohex
  301. oidc
  302. opam-0install
  303. opam-file-format >= "2.1.1"
  304. opentelemetry >= "0.6"
  305. opentelemetry-client-cohttp-lwt >= "0.6"
  306. opentelemetry-client-ocurl >= "0.6"
  307. opentelemetry-cohttp-lwt >= "0.6"
  308. opentelemetry-lwt >= "0.6"
  309. opium >= "0.15.0"
  310. opium-graphql
  311. opium-testing
  312. opium_kernel
  313. orewa
  314. ortac-core
  315. osx-acl
  316. osx-attr
  317. osx-cf
  318. osx-fsevents
  319. osx-membership
  320. osx-mount
  321. osx-xattr
  322. otoggl
  323. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  324. owl-base < "0.5.0"
  325. owl-ode >= "0.1.0" & != "0.2.0"
  326. owl-symbolic
  327. passmaker
  328. patch
  329. pbkdf
  330. pecu >= "0.2"
  331. pf-qubes
  332. pg_query >= "0.9.6"
  333. pgx >= "1.0"
  334. pgx_unix >= "1.0"
  335. pgx_value_core
  336. pgx_value_ptime < "2.2"
  337. phylogenetics
  338. piaf
  339. polyglot
  340. polynomial
  341. ppx_blob >= "0.3.0"
  342. ppx_deriving_cmdliner
  343. ppx_deriving_rpc
  344. ppx_deriving_yaml
  345. ppx_graphql >= "0.2.0"
  346. ppx_inline_alcotest
  347. ppx_parser
  348. ppx_protocol_conv >= "5.0.0"
  349. ppx_protocol_conv_json >= "5.0.0"
  350. ppx_protocol_conv_jsonm >= "5.0.0"
  351. ppx_protocol_conv_msgpack >= "5.0.0"
  352. ppx_protocol_conv_xml_light >= "5.0.0"
  353. ppx_protocol_conv_xmlm
  354. ppx_protocol_conv_yaml >= "5.0.0"
  355. ppx_repr < "0.4.0"
  356. ppx_subliner
  357. ppx_units
  358. ppx_yojson >= "1.1.0"
  359. pratter
  360. prc
  361. preface
  362. pretty_expressive
  363. prettym
  364. proc-smaps
  365. producer < "0.2.0"
  366. progress < "0.2.0"
  367. prom
  368. prometheus < "1.2"
  369. prometheus-app
  370. protocell
  371. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  372. protocol-9p-unix
  373. psq
  374. qcheck >= "0.18"
  375. qcheck-alcotest
  376. qcheck-core >= "0.18"
  377. quickjs
  378. radis
  379. randii
  380. reason-standard
  381. reparse >= "2.0.0" & < "3.0.0"
  382. reparse-unix < "2.1.0"
  383. resp
  384. resp-unix >= "0.10.0"
  385. rfc1951 < "1.0.0"
  386. routes < "2.0.0"
  387. rpc >= "7.1.0"
  388. rpclib >= "7.1.0"
  389. rpclib-async
  390. rpclib-lwt >= "7.1.0"
  391. rubytt
  392. SZXX >= "4.0.0"
  393. salsa20
  394. salsa20-core
  395. sanddb >= "0.2"
  396. scaml >= "1.5.0"
  397. scrypt-kdf
  398. secp256k1 >= "0.4.1"
  399. secp256k1-internal
  400. semver >= "0.2.1"
  401. sendmail
  402. sendmail-lwt
  403. sendmsg
  404. server-reason-react
  405. session-cookie
  406. session-cookie-async
  407. session-cookie-lwt
  408. sherlodoc
  409. slug
  410. sodium-fmt
  411. spin >= "0.6.0"
  412. squirrel
  413. ssh-agent
  414. ssl >= "0.6.0"
  415. stramon-lib
  416. styled-ppx
  417. syslog-rfc5424
  418. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "7.0.0"
  419. tdigest < "2.1.0"
  420. terminal_size >= "0.1.1"
  421. terminus
  422. terminus-cohttp
  423. terminus-hlc
  424. terml
  425. textrazor
  426. tezos-base-test-helpers < "13.0"
  427. tezos-bls12-381-polynomial
  428. tezos-client-base < "12.0"
  429. tezos-crypto >= "8.0" & < "9.0"
  430. tezos-lmdb
  431. tezos-plompiler = "0.1.3"
  432. tezos-plonk = "0.1.3"
  433. tezos-signer-backends >= "8.0" & < "13.0"
  434. tezos-stdlib >= "8.0" & < "12.0"
  435. tezos-test-helpers < "12.0"
  436. tftp
  437. timedesc
  438. timere
  439. tls >= "0.12.0"
  440. toc
  441. topojson
  442. topojsone
  443. transept
  444. twostep
  445. type_eq
  446. type_id
  447. typebeat
  448. typeid >= "1.0.1"
  449. tyre >= "0.4"
  450. tyxml >= "4.0.0"
  451. tyxml-jsx
  452. tyxml-ppx >= "4.3.0"
  453. tyxml-syntax
  454. uecc
  455. ulid
  456. universal-portal
  457. unix-dirent
  458. unix-errno >= "0.3.0"
  459. unix-fcntl >= "0.3.0"
  460. unix-sys-resource
  461. unix-sys-stat
  462. unix-time
  463. unstrctrd
  464. user-agent-parser
  465. uspf
  466. uspf-lwt
  467. uspf-unix
  468. utop >= "2.13.0"
  469. validate
  470. validator
  471. vercel
  472. vpnkit
  473. wcwidth
  474. websocketaf
  475. x509 >= "0.7.0"
  476. xapi-rrd >= "1.8.2"
  477. xapi-stdext-date
  478. xapi-stdext-encodings
  479. xapi-stdext-std >= "4.16.0"
  480. yaml < "3.2.0"
  481. yaml-sexp
  482. yocaml
  483. yocaml_yaml
  484. yojson >= "1.6.0"
  485. yuscii >= "0.3.0"
  486. zar
  487. zed >= "3.2.2"
  488. zlist < "0.4.0"

Conflicts

None

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct