Property based testing

From theory to practice

Nicolas DUBIEN

2014 - 2017
C++ at Murex
 
2017 -
TypeScript and C# at Criteo
 
dubzzz ndubien

Theory

Face the unpredictable

Practice

Let's break some stuff

Theory

Face the unpredictable

Coding safely

Static code analysis

interpreter, compiler, linter…

Example based testing

automated tests, qa tests…

Proved code

 

Crash under random

fuzzing, monkey test…

Property based testing in few words

We want to test properties instead of isolated cases.

for all (x, y, ...)
generate random inputs based on specified generators
such that precondition(x, y, ...) holds
check pre-conditions - failure? go back to previous
property(x, y, ...) is true
run the test - failure? shrink

🔁 Run it 100 times 🔁

Example

Let’s suppose we want to test isSubstring

for all (x, y, ...)
such that precondition(x, y, ...) holds
property(x, y, ...) is true

for all a, b, c strings
b is a substring of a + b + c

aze

a

+

 

rt

b

+

 

y

c

Shrink

Randomly generated data can be very complex
Shrink produce smaller entries derived from complex ones

for all a, b, c strings
b is a substring of a + b + c

e漢}@🐱z’

a

+

 

eiiz

b

+

 

漢null¤

c

Shrinking into action

{a: "e漢}@🐱z’", b: "eiiz", c: "漢null¤"}

{a: "", b: "eiiz", c: "漢null¤"}

{a: "", b: "", c: "漢null¤"}

{a: "", b: "ei", c: "漢null¤"}

{a: "", b: "eii", c: "漢null¤"}

{a: "", b: "ii", c: "漢null¤"}

{a: "", b: "i", c: "漢null¤"}

{a: "", b: "ii", c: ""}

Benefits

Reproductible

Straight to corner cases

Use the scope of all possible inputs

Shrink the input in case of failure

Don't...

Expect to test the exact result

Rewrite your implementation in the test

Replace your examples

Use too complex pre-conditions

References

js-yaml

query-string

left-pad

Practice

Let's break some stuff

dubzzz/fast-check fast-check

Easy inputs

trekhleb/javascript-algorithms

Complex objects

sindresorhus/query-string

UI test

2048 game

Easy inputs

trekhleb/javascript-algorithms

Initial project setup

Project under tests

Algorithm:

isSubstring based on Knuth-Morris-Pratt / Rabin-Karp (2)

for all a, b, c strings
b is a substring of a + b + c


test("should detect the substring", () =>
  fc.assert(
    fc.property(
      fc.fullUnicodeString(),
      fc.fullUnicodeString(),
      fc.fullUnicodeString(),
      (a, b, c) => knuth(a + b + c, b) !== -1
    )
  ));
                    

Only covered the truthy state

Both algorithms return the position of the match as an output

inviter

a

+

 

vite

b

knuth(a, b) == 2
a.substring(2, 2 + b.length) == b

for all a, b strings and idx = a.indexOf(b)
is either -1 or position of the match


test("should send back a valid start", () =>
  fc.assert(
    fc.property(fc.fullUnicodeString(), fc.fullUnicodeString(),
      (a, b) => {
        const idx = knuth(a, b);
        return idx === -1 || a.substr(idx, b.length) === b;
      }
    )
  ));
                    

Complex objects

sindresorhus/query-string

Algorithm:

Serialization and Deserialization of query parameters

Source: query-string@6.0.0

More: https://github.com/sindresorhus/query-string

.stringify(object, [options])

.parse(string, [options])

for all obj query string object, opts options
parse(stringify(obj, opts), opts) is obj

UI test

2048 game

We are already able to:

  • produce random inputs: integers, arrays, ...
  • play our tests randomly
  • shrink to the minimal case

We need to define commands:

  • check if it can be applied to current state
  • apply it

The framework can produce arrays of commands and shrink them in case of problem

https://dubzzz.github.io/scala-2048/

Example: Redo

  • Can only be executed if a move has just been canceled
  • Should bring the user back to the state she/he was before canceling the move

Example: Undo

  • Can only be executed if there was at least one move played
  • Should bring the user back to the state she/he was before the move

Questions?

dubzzz/fast-check fast-check
fast-check