package qcheck-core

  1. Overview
  2. Docs
Core qcheck library


Dune Dependency







test property quickcheck

Published: 16 Feb 2021



= QCheck
:toc: macro
:toclevels: 4
:source-highlighter: pygments

QuickCheck inspired property-based testing for OCaml, and combinators to
generate random values to run tests on.

The documentation can be found[here].
This library spent some time in[qtest], but is now
standalone again!

To construct advanced random generators, the following libraries might be
of interest:

- @gasche's[generator library]

Jan Midtgaard has[a lecture] about
property-based testing that relies on QCheck.


image::[alt="Build Status", link=""]

== Use

See the documentation. I also wrote[a blog post] that explains
how to use it and some design choices; however, be warned that the API
changed in lots of small ways (in the right direction, I hope) so the code
will not work any more.
<<examples>> is an updated version of the blog post's examples.

== Build

    $ make

You can use opam:

    $ opam install qcheck

== License

The code is now released under the BSD license.

== An Introduction to the Library

First, let's see a few tests. Let's open a toplevel (e.g. utop)
and type the following to load QCheck:

#require "qcheck";;

NOTE: alternatively, it is now possible to locally do: `dune utop src`
to load `qcheck`.

=== List Reverse is Involutive

We write a random test for checking that `List.rev (List.rev l) = l` for
any list `l`:

let test =
  QCheck.Test.make ~count:1000 ~name:"list_rev_is_involutive"
   QCheck.(list small_nat)
   (fun l -> List.rev (List.rev l) = l);;

(* we can check right now the property... *)
QCheck.Test.check_exn test;;

In the above example, we applied the combinator `list` to
the random generator `small_nat` (ints between 0 and 100), to create a
new generator of lists of random integers. These builtin generators
come with printers and shrinkers which are handy for outputting and
minimizing a counterexample when a test fails.

Consider the buggy property `List.rev l = l`:

let test =
  QCheck.Test.make ~count:1000 ~name:"my_buggy_test"
   QCheck.(list small_nat)
   (fun l -> List.rev l = l);;

When we run this test we are presented with a counterexample:

# QCheck.Test.check_exn test;;
QCheck.Test.Test_fail ("my_buggy_test", ["[0; 1] (after 23 shrink steps)"]).

In this case QCheck found the minimal counterexample `[0;1]` to the property
`List.rev l = l` and it spent 23 steps shrinking it.

Now, let's run the buggy test with a decent runner that will print the results
nicely (the exact output will change at each run, because of the random seed):

# QCheck_runner.run_tests [test];;

--- Failure --------------------------------------------------------------------

Test my_buggy_test failed (10 shrink steps):

[0; 1]
failure (1 tests failed, 0 tests errored, ran 1 tests)
- : int = 1

For an even nicer output `QCheck_runner.run_tests` also accepts an optional
parameter `~verbose:true`.

=== Mirrors and Trees

`QCheck` provides many useful combinators to write
generators, especially for recursive types, algebraic types,
and tuples.

Let's see how to generate random trees:

type tree = Leaf of int | Node of tree * tree

let leaf x = Leaf x
let node x y = Node (x,y)

let tree_gen = QCheck.Gen.(sized @@ fix
  (fun self n -> match n with
    | 0 -> map leaf nat
    | n ->
        [1, map leaf nat;
         2, map2 node (self (n/2)) (self (n/2))]

(* generate a few trees, just to check what they look like: *)
QCheck.Gen.generate ~n:20 tree_gen;;

let arbitrary_tree =
  let open QCheck.Iter in
  let rec print_tree = function
    | Leaf i -> "Leaf " ^ (string_of_int i)
    | Node (a,b) -> "Node (" ^ (print_tree a) ^ "," ^ (print_tree b) ^ ")"
  let rec shrink_tree = function
    | Leaf i -> i >|= leaf
    | Node (a,b) ->
      of_list [a;b]
      (shrink_tree a >|= fun a' -> node a' b)
      (shrink_tree b >|= fun b' -> node a b')
  QCheck.make tree_gen ~print:print_tree ~shrink:shrink_tree;;


Here we write a generator of random trees, `tree_gen`, using
the `fix` combinator. `fix` is *sized* (it is a function from `int` to
a random generator; in particular for size 0 it returns only leaves).
The `sized` combinator first generates a random size, and then applies
its argument to this size.

Other combinators include monadic abstraction, lifting functions,
generation of lists, arrays, and a choice function.

Then, we define `arbitrary_tree`, a `tree QCheck.arbitrary` value, which
contains everything needed for testing on trees:

- a random generator (mandatory), weighted with `frequency` to
  increase the chance of generating deep trees
- a printer (optional), very useful for printing counterexamples
- a *shrinker* (optional), very useful for trying to reduce big
  counterexamples to small counterexamples that are usually
  more easy to understand.

The above shrinker strategy is to

- reduce the integer leaves, and
- substitute an internal `Node` with either of its subtrees or
  by splicing in a recursively shrunk subtree.

A range of combinators in `QCheck.Shrink` and `QCheck.Iter` are available
for building shrinking functions.

We can write a failing test using this generator to see the
printer and shrinker in action:

let rec mirror_tree (t:tree) : tree = match t with
  | Leaf _ -> t
  | Node (a,b) -> node (mirror_tree b) (mirror_tree a);;

let test_buggy =
  QCheck.Test.make ~name:"buggy_mirror" ~count:200
    arbitrary_tree (fun t -> t = mirror_tree t);;

QCheck_runner.run_tests [test_buggy];;

This test fails with:


--- Failure --------------------------------------------------------------------

Test mirror_buggy failed (6 shrink steps):

Node (Leaf 0,Leaf 1)
failure (1 tests failed, 0 tests errored, ran 1 tests)
- : int = 1

With the (new found) understanding that mirroring a tree
changes its structure, we can formulate another property
that involves sequentializing its elements in a traversal:

let tree_infix (t:tree): int list =
  let rec aux acc t = match t with
    | Leaf i -> i :: acc
    | Node (a,b) ->
      aux (aux acc b) a
  aux [] t;;

let test_mirror =
  QCheck.Test.make ~name:"mirror_tree" ~count:200
    (fun t -> List.rev (tree_infix t) = tree_infix (mirror_tree t));;

QCheck_runner.run_tests [test_mirror];;


=== Preconditions

The functions `QCheck.assume` and `QCheck.(==>)` can be used for
tests with preconditions.
For instance, `List.hd l :: l = l` only holds for non-empty lists.
Without the precondition, the property is false and will even raise
an exception in some cases.

let test_hd_tl =
    (list int) (fun l ->
      assume (l <> []);
      l = List.hd l :: l));;

QCheck_runner.run_tests [test_hd_tl];;

=== Long tests

It is often useful to have two version of a testsuite: a short one that runs
reasonably fast (so that it is effectively run each time a projet is built),
and a long one that might be more exhaustive (but whose running time makes it
impossible to run at each build). To that end, each test has a 'long' version.
In the long version of a test, the number of tests to run is multiplied by
the `~long_factor` argument of `QCheck.Test.make`.

=== Runners

The module `QCheck_runner` defines several functions to run tests, including
compatibility with `OUnit`.
The easiest one is probably `run_tests`, but if you write your tests in
a separate executable you can also use `run_tests_main` which parses
command line arguments and exits with `0` in case of success,
or an error number otherwise.

=== Integration within OUnit[OUnit] is a popular unit-testing framework
for OCaml.
QCheck provides a sub-library `qcheck-ounit` with some helpers, in `QCheck_ounit`,
to convert its random tests into OUnit tests that can be part of a wider

let passing =
  QCheck.Test.make ~count:1000
    QCheck.(list small_nat)
    (fun l -> List.rev (List.rev l) = l);;

let failing =
  QCheck.Test.make ~count:10
    QCheck.(list small_nat)
    (fun l -> l = List.sort compare l);;

let _ =
  let open OUnit in
    ("tests" >::: QCheck_ounit.to_ounit_test [passing; failing])


NOTE: the package `qcheck` contains the module `QCheck_runner`
which contains both custom runners and OUnit-based runners.

=== Integration within alcotest[Alcotest] is a simple and colorful test framework for
OCaml. QCheck now provides a sub-library `qcheck-alcotest` to
easily integrate into an alcotest test suite:


let passing =
  QCheck.Test.make ~count:1000
    QCheck.(list small_int)
    (fun l -> List.rev (List.rev l) = l);;

let failing =
  QCheck.Test.make ~count:10
    QCheck.(list small_int)
    (fun l -> l = List.sort compare l);;

let () =
  let suite = QCheck_alcotest.to_alcotest
      [ passing; failing]
  in "my test" [
    "suite", suite


=== Integration within Rely[Rely] is a Jest-inspire native reason testing framework.
@reason-native/qcheck-rely is available via NPM and provides matchers for the easy
use of qCheck within Rely.

[source, Reason]
open TestFramework;
open QCheckRely;

let {describe} = extendDescribe(QCheckRely.Matchers.matchers);

describe("qcheck-rely", ({test}) => {
  test("passing test", ({expect}) => {
    let passing =
        l =>
        List.rev(List.rev(l)) == l
  test("failing test", ({expect}) => {
    let failing =
        ~count=10, ~name="fail_sort_id", QCheck.(list(small_int)), l =>
        l == List.sort(compare, l)



=== Compatibility notes

Starting with 0.9, the library is split into several components:

- `qcheck-core` depends only on unix and bytes. It contains the module
  `QCheck` and a `QCheck_base_runner` module with our custom runners.
- `qcheck-ounit` provides an integration layer for `OUnit`
- `qcheck` provides a compatibility API with older versions of qcheck,
  using both `qcheck-core` and `qcheck-ounit`.
  It provides `QCheck_runner` which is similar to older versions and contains
  both custom and Ounit-based runners.
- `qcheck-alcotest` provides an integration layer with `alcotest`

Normally, for contributors,
`opam pin` will pin all these packages.

Dependencies (4)

  1. ocaml >= "4.03.0"
  2. base-unix
  3. base-bytes
  4. dune

Dev Dependencies (1)

  1. odoc with-doc

Used by (20)

  1. containers >= "3.9" & < "3.13"
  2. containers-data >= "3.9" & < "3.13"
  3. containers-thread >= "3.9"
  4. goblint >= "1.1.1" & < "2.2.1"
  5. iter >= "1.6"
  6. logtk >= "1.5.1"
  7. lru >= "0.3.0"
  8. octez-libs
  9. octez-shell-libs
  10. ortac-qcheck-stm
  11. osnap >= "0.3.0"
  12. preface < "1.0.0"
  13. psq >= "0.1.1"
  14. qcheck = "0.17"
  15. qcheck-alcotest = "0.17"
  16. qcheck-ounit = "0.17"
  17. tezos-proxy >= "17.1"
  18. tezos-shell-services-test-helpers < "12.0" | >= "13.0"
  19. tezos-webassembly-interpreter < "15.0"
  20. tiny_httpd >= "0.13"

Conflicts (1)

  1. ounit < "2.0"

Innovation. Community. Security.