Improve your software quality with Property-Based Testing

6 min readApr 6, 2020

Write bulletproof code with PBT

Before talking about PBT, let’s talk about the different test technics we use to test the correctness of our code. Those technics are represented on 2 axis :

Input scope covered: Do we cover the full scope of possible inputs ?
Feature compliance: Does the developed feature is compliant with what is expected ?

When we write Unit tests, we often focus on examples especially when using approaches like Specification by examples. Examples are really good to understand requirements :

Given (x, y, ...) When I [call the subject under test] with (x, y, ...) Then I expect this (output)

The limitation of this approach is that we will only focus on the input scope we identified. It is where PBT comes at our rescue.

Imagine an approach as powerful as Example based testing in terms of feature compliance checking but covering also a much more important numbers of inputs. That’s the promise of PBT.

What is PBT ?

Property-based testing is generative testing. You do not supply specific example inputs with expected outputs.

Instead, you define properties about your feature requirements and use a generative-testing engine to create randomized inputs to ensure the defined properties are correct.

Property-based tests are designed to test the aspects of a property that should always be true. They allow for a range of inputs to be programmed and tested within a single test, rather than having to write a different test for every value that you want to test.

A property looks like this :

for all (x, y, ...) such that property (x, y, ...) is satisfied

It’s used a lot in the functional world where we favor the writing of pure functions (same output for a given output without side effects).

Let’s take an example

Imagine we write an addition function that add 2 integers together. With Example-Based Testing we would write tests like this :

Given (2, 2) When I add them Then I expect 4

With this test we cover 100% of the code but what about edge cases ? Add negative numbers together for example.

With PBT we would identify the properties of the addition operation :

Commutativity : the parameter order does not matter

for all (int x, int y) such that (add(x, y) equals add(y, x)) is satisfied

Associativity : “add 1” twice is the same as doing “add 2”

for all (int x) such that (add(add(x, 1), 1) equals add(x, 2)) is satisfied

Identity : add 0 does nothing

for all (int x) such that (add(x, 0) equals x) is satisfied

If we check those properties to all inputs we are confident the implementation is correct. We do not check specific values in output anymore we check the properties.

A word on QuickCheck

The first tool used to do PBT was QuickCheck (in Haskell) and in this hands-on I will talk only for the behavior of QuickCheck. How does a PBT tool work ?

You define a Property : a function that returns a boolean
You pass it to a checker that will generate random inputs (through the generator in QuickCheck)
A shrinker will attempt to shrink the input sequence to the smallest possible that will reproduce the error. The smaller the input, the easier it is to reproduce and fix.
The Runner will use the whole to run your predicate (Property) with the generated

As soon as there is one value which yields false, the property is said to be falsified, and checking is aborted.

How to implement PBT in Java ?

An implementation of QuickCheck is available for Java : junit-quickcheck

Add the dependency to your pom :

Define your properties

On top of your PBT test classes you need to specify the Runner to be used : JUnitQuickCheck.class
On each Property, add the annotation @Property

Here it is we just have created our first PBT class to test our Calculator.add function.

Junit-quickcheck

It supports a lot of types by default :

Configure generator

You can specify ranges for your inputs with @Range :

Or you can use assumptions to restrain those values :

Generate complex inputs :

To instantiate complex objects in input of your functions, you can simply create custom Generators :

Default behaviors :

By default it will verify a property in “sampling” mode, it generates 100 tuples of random values for the parameter list of a property, and verifies the property against each of the tuples.

To change the number of generated values, use the trials attribute of the @Property annotation.

PBT in real life

Imagine we have an Account class which defines the behavior of a withdraw.

We would like to test the feature compliance. To do so we identify the properties :

λ The account balance should be decremented of the withdrawal amount when the account balance is sufficient or the overdraft is authorized

λ The client must not be allowed to withdraw when the withdraw amount is over the max withdrawal amount of the account OR the account balance is insufficient and overdraft is not authorized for the account

Create custom generators

To write PBTs we need to be able to generate random Inputs (withdraw) and random accounts for a given property. To do so we simply create 2 generators :

Those generators are rally dumb, we just want them to generate complex data structure by using random values in the constructors.

Use the generators and assumptions

To use a custom generator you can pass it through the @From annotation. We need to use assumptions to declare the conditions under the properties hold.

I encapsulate those assumptions in small functions to understand quickly what are the conditions for the properties.

Another lib — vavr-test

You can use another java lib to write PBTs : vavr-test

It allows you to write PBT in more functional way. If we write the same properties for the addition it would look like this :

With vavr-test you don’t need to specify a given Runner nor Property annotation. You specify Property in your test.

Quickcheck in your favorite language

Source : https://hypothesis.works/articles/quickcheck-in-every-language/

Benefits of PBT

PBTs are more general than Example based testing : 1 PBT can replace many example-based tests
PBTs can reveal edge cases : through the usage of random values (Nulls, negative numbers, weird strings,…)
PBTs ensures deep understanding of the business : to be able to identify properties we must deeply understand business invariants