Property based testing is another way to test programs. Instead of relying on hard-coded inputs and outputs, it checks characteristics of the output given the whole range of possible inputs.

Coverage more >

By nature, property based testing puts less constraints on the inputs. As a consequence, the scope of covered inputs is much higher and can lead to unexplored code paths

Reproducible more >

Runs are seeded and fully reproducible

Shrink more >

In case of failure, the framework is responsible to come with a minimal failing case easy to troubleshoot

Getting started

Property based testing frameworks check the truthfulness of properties

Add fast-check to your project:

npm install --save-dev fast-check

Example of integration with Jest or Mocha:

import * as fc from 'fast-check';

const sort = arr => arr.slice(0).sort((a,b) => a-b);

test('should order elements from the smallest to the highest', () => fc.assert(
	fc.property(
		fc.array(fc.integer()),
		arr => {
			const sortedArr = sort(arr);
			for (let idx = 1 ; idx < sortedArr.length ; ++idx)
				if (sortedArr[idx - 1] <= sortedArr[idx])
					return false;
			return true;
		}
	)
));

In a nutshell:

  • fc.assert runs the property
  • fc.property defines the property
  • fc.array(fc.integer()) defines the inputs the framework has to generate
  • arr => { ... } checks the output against the generated value

Coverage

Discover uncovered code paths

Let's consider a simple serialize function, offering its users the ability to set custom settings to tweak the resulting string

function serialize<T>(instance: T, params: Parameters): string { /* code */ }

In the scenario above, an exhaustive test would have to run throughout all the possible combinations of params with many possible instance. An alternative to such example-based tests would be to rely on properties. It would offer an exhautive coverage of the possible inputs. Given we also have a deserialize function we would simply write:

test('should be able to read itself', () => fc.assert(
	fc.property(
		fc.jsonObject(),
		fc.record({
			crlf: fc.boolean(),
			indent: fc.boolean(),
			sortKeys: fc.boolean()
		}, { withDeletedKeys: true }),
		(instance, params) => {
			expect(deserialize(serialize(instance, params))).toEqual(instance);
		}
	)
));

Such property was able to detect bugs in both js-yaml and query-string

Reproducible

Replay the same test after a fix

Whenever fast-check detects a problem, it will print an error message containing the settings required to replay the very same test

Error: Property failed after 1 tests (seed: 1527423434693, path: "0:0:0"): ["","",""]
Shrunk 2 time(s)
Got error: Property failed by returning false

Given this message the test can easily be replayed by using:

test('the failing test', () => fc.assert(
	fc.property(
		// some arbitraries... 
		// check method
	), { // seed and path taken from the error message
		seed: 1527423434693,
		path: "0:0:0"
	}
));

Setting both seed and path will replay the test starting with the latest failing case. If you want to replay the whole test suite, you should only set the seed

Shrink

Get a better understanding of the errors

Whenever a test fails, fast-check will try to ease your investigations by providing you with the minimal failing case. Sometimes it is not enough and having all the failures might help. For this precise need, fc.assert accepts an optional configuration: verbose: true.

test('the failing test', () => fc.assert(
	fc.property(
		// some arbitraries... 
		// check method
	), { verbose: true }
));

With this setting enabled the error message will look like:

Error: Property failed after 1 tests (seed: 1527423434693, path: 0:0:0): ["","",""]
Shrunk 2 time(s)
Got error: Property failed by returning false

Encountered failures were:
- ["","JeXPqIQ6",">q"]
- ["","",">q"]
- ["","",""]

Fork me on GitHub