Reviewing Property-Based Tests

Evaluate quality of existing property-based tests and suggest improvements.

Quick Reference

Quality Issues

Issue: Tautological Properties (CRITICAL)

Properties that are always true regardless of implementation.

# BAD - compares function to itself
@given(st.lists(st.integers()))
def test_sort_tautology(xs):
    assert sorted(xs) == sorted(xs)  # Always true!

# BAD - tests nothing about the function
@given(st.integers())
def test_useless(x):
    result = compute(x)
    assert result == result  # Always true!

Detection: Assertions comparing same expression, or not using function result meaningfully.

Issue: Vacuous Tests (CRITICAL)

Tests where assumptions filter out most/all inputs.

# VACUOUS - impossible condition
@given(st.integers())
def test_vacuous(x):
    assume(x > 100)
    assume(x < 50)  # Impossible!
    assert compute(x) > 0

# VACUOUS - overly restrictive
@given(st.integers())
def test_too_filtered(x):
    assume(x == 42)  # Only tests one value!
    assert compute(x) == expected

Detection: Multiple assume() calls, assume with very narrow conditions.

Issue: Weak Properties (HIGH)

Properties that only test minimal guarantees.

# WEAK - only tests no crash
@given(st.text())
def test_only_no_crash(s):
    process(s)  # No assertion at all

# WEAK - only tests type
@given(st.integers())
def test_only_type(x):
    assert isinstance(compute(x), int)

Detection: Tests without assertions, or only isinstance/type checks.

Issue: Reimplementing the Function (HIGH)

# BAD - just reimplements the logic
@given(st.integers(), st.integers())
def test_reimplements(a, b):
    assert add(a, b) == a + b  # Tests nothing if add() is just a + b

Detection: Test assertion contains same logic as function under test.

Issue: Poor Input Coverage (MEDIUM)

# NARROW - misses edge cases
@given(st.integers(min_value=1, max_value=10))
def test_narrow_range(x):
    assert compute(x) >= 0  # What about 0? Negatives? Large values?

# MISSING - no edge case examples
@given(st.lists(st.integers()))
def test_no_explicit_edges(xs):
    # Should include @example([]) @example([1]) etc.
    assert len(sort(xs)) == len(xs)

Issue: Missing Stronger Properties (MEDIUM)

# EXISTS - but could be stronger
@given(st.lists(st.integers()))
def test_sort_length(xs):
    assert len(sort(xs)) == len(xs)
# MISSING: ordering property, element preservation

Issue: Poor Settings (LOW)

# TOO FEW - may miss bugs
@settings(max_examples=5)
def test_few_examples(x): ...

# NO DEADLINE - may hang in CI
@given(expensive_strategy())
def test_no_deadline(x): ...  # Could timeout

Review Process

1. Locate Property-Based Tests

Search using library-specific patterns:

Python/Hypothesis:

rg "@given\\(" --type py
rg "from hypothesis import" --type py

JavaScript/fast-check:

rg "fc\\.(assert|property)" --type js --type ts

Rust/proptest:

rg "proptest!" --type rust

2. Analyze Each Test

Check for issues above, starting with critical then high severity.

3. Evaluate Shrinking Quality

Will tests shrink to minimal counterexamples? Complex strategies may produce hard-to-debug failures.

4. Check for Flakiness Potential

• Non-determinism in code under test
• Time-dependent assertions
• Global state dependencies
• Floating point comparisons without tolerance

5. Suggest Stronger Properties

Compare against property catalog - are stronger properties available but not tested?

Test Health Score

Mutation Testing Verification

Suggest specific mutations to verify tests catch bugs:

To verify test_sort catches bugs:

1. Return input unchanged: `return xs`
   - Should fail: test_ordering

2. Drop last element: `return sorted(xs)[:-1]`
   - Should fail: test_length_preserved

3. Reverse order: `return sorted(xs, reverse=True)`
   - Should fail: test_ordering

Quality Checklist

For each test, verify:

• Not tautological (assertion doesn't compare same expression)
• Strong assertion (not just "no crash")
• Not vacuous (inputs not over-filtered)
• Good coverage (edge cases via @example)
• No reimplementation of function logic
• Appropriate settings for context
• Good shrinking potential
• Deterministic (no flakiness risk)

Red Flags

• Marking tautologies as "fine": assert x == x is NEVER a valid test
• Accepting "no crash" as sufficient: Always push for stronger properties
• Ignoring vacuous tests: Tests with contradictory assume() provide false confidence
• Not checking for reimplementation: assert add(a,b) == a + b tests nothing if that's how add is implemented