Property based testing is another way to test programs. Instead of relying on hard-coded inputs and outputs, it checks characteristics of the output given the whole range of possible inputs.
By nature, property based testing puts less constraints on the inputs. As a consequence, the scope of covered inputs is much higher and can lead to unexplored code paths
Runs are seeded and fully reproducible
In case of failure, the framework is responsible to come with a minimal failing case easy to troubleshoot
Property based testing frameworks check the truthfulness of properties
Add fast-check to your project:
npm install --save-dev fast-check
Example of integration with Jest or Mocha:
import * as fc from 'fast-check';
const sort = arr => arr.slice(0).sort((a,b) => a-b);
test('should order elements from the smallest to the highest', () => fc.assert(
fc.property(
fc.array(fc.integer()),
arr => {
const sortedArr = sort(arr);
for (let idx = 1 ; idx < sortedArr.length ; ++idx)
if (sortedArr[idx - 1] <= sortedArr[idx])
return false;
return true;
}
)
));
In a nutshell:
Discover uncovered code paths
Let's consider a simple serialize function, offering its users the ability to set custom settings to tweak the resulting string
function serialize<T>(instance: T, params: Parameters): string { /* code */ }
In the scenario above, an exhaustive test would have to run throughout all the possible combinations of params with many possible instance. An alternative to such example-based tests would be to rely on properties. It would offer an exhautive coverage of the possible inputs. Given we also have a deserialize function we would simply write:
test('should be able to read itself', () => fc.assert(
fc.property(
fc.jsonObject(),
fc.record({
crlf: fc.boolean(),
indent: fc.boolean(),
sortKeys: fc.boolean()
}, { withDeletedKeys: true }),
(instance, params) => {
expect(deserialize(serialize(instance, params))).toEqual(instance);
}
)
));
Such property was able to detect bugs in both js-yaml and query-string
Replay the same test after a fix
Whenever fast-check detects a problem, it will print an error message containing the settings required to replay the very same test
Error: Property failed after 1 tests (seed: 1527423434693, path: "0:0:0"): ["","",""]
Shrunk 2 time(s)
Got error: Property failed by returning false
Given this message the test can easily be replayed by using:
test('the failing test', () => fc.assert(
fc.property(
// some arbitraries...
// check method
), { // seed and path taken from the error message
seed: 1527423434693,
path: "0:0:0"
}
));
Setting both seed and path will replay the test starting with the latest failing case. If you want to replay the whole test suite, you should only set the seed
Get a better understanding of the errors
Whenever a test fails, fast-check will try to ease your investigations by providing you with the minimal failing case. Sometimes it is not enough and having all the failures might help. For this precise need, fc.assert accepts an optional configuration: verbose: true.
test('the failing test', () => fc.assert(
fc.property(
// some arbitraries...
// check method
), { verbose: true }
));
With this setting enabled the error message will look like:
Error: Property failed after 1 tests (seed: 1527423434693, path: 0:0:0): ["","",""]
Shrunk 2 time(s)
Got error: Property failed by returning false
Encountered failures were:
- ["","JeXPqIQ6",">q"]
- ["","",">q"]
- ["","",""]