Interpreting Property-Based Test Failures

How to analyze failures and determine if they represent genuine bugs.

The Self-Reflection Problem

Property-based testing generates many failing examples. Not all failures are bugs:

• Test bugs: Property is wrong, strategy generates invalid inputs
• Ambiguous specs: Behavior undefined for edge cases
• Genuine bugs: Code violates documented guarantees

Before reporting a bug, validate the failure through systematic analysis.

Failure Analysis Workflow

1. Reproduce with Minimal Example

Start with the shrunk failing input from the test output.

# Hypothesis provides the minimal failing case
# Falsifying example: test_normalize(s='\\x00')

# Create standalone reproducer
def test_reproduce():
    s = '\\x00'
    result = normalize(normalize(s))
    assert result == normalize(s)  # Fails

Verify the failure is consistent, not flaky.

2. Ground the Property

Before assuming a bug, verify your property against authoritative sources:

Example grounding check:

def normalize(s: str) -> str:
    """Normalize a string to NFC form.

    Args:
        s: Input string (any unicode)

    Returns:
        NFC-normalized string
    """

The docstring says "any unicode" - so null bytes should be valid input. The property is correctly grounded.

3. Check Strategy Realism

Does the strategy generate inputs the function should actually handle?

Red flags:

• Generating inputs outside documented domain
• Missing constraints that real callers would have
• Overly aggressive size/complexity

Questions to ask:

• Would real code pass this input?
• Does the docstring exclude this case?
• Is this a precondition violation, not a bug?

4. Classify the Failure

5. Decide Action

• Test bug → Fix the property or strategy, don't report
• Ambiguous spec → Open discussion issue, not bug report
• Genuine bug → Report with minimal reproducer and evidence

Property Grounding Checklist

Before reporting a failure as a bug, verify:

• Property matches documented return type
• Property matches docstring guarantees
• Input is within documented domain (preconditions met)
• No assume() filtering out the failing case inappropriately
• Checked existing tests don't contradict your property
• Behavior contradicts docs, not just expectations

Bug Report Template

When confident the failure is a genuine bug:

## Summary
[One-line description of the bug]

## Minimal Reproducing Example
```python
# Shrunk by Hypothesis
from mylib import affected_function

def test_bug():
    # Minimal failing input
    result = affected_function('\\x00')
    # Expected vs actual
    assert result >= 0  # Fails: got -1

Expected Behavior

According to [docstring/spec/docs], the function should:

• [Specific guarantee that was violated]

Actual Behavior

• [What actually happened]

Evidence

• Docstring states: "[relevant quote]"
• Type signature promises: -> PositiveInt

Environment

• Library version: X.Y.Z
• Python version: 3.X
• Platform: [OS]

## Real-World Failure Patterns

### Numerical Instability

**Symptom**: Distribution function returns negative probability.

```python
@given(st.floats(min_value=0, max_value=1e308))
def test_probability_non_negative(x):
    prob = compute_probability(x)
    assert prob >= 0  # Fails for x=1e-320

Grounding check: Docstring says "returns probability in [0, 1]".

Classification: Genuine bug - documented guarantee violated.

Iterator Off-by-One

Symptom: Iterator skips elements or yields extra.

@given(st.lists(st.integers()))
def test_iterator_yields_all(xs):
    result = list(custom_iterator(xs))
    assert result == xs  # Fails: missing last element

Grounding check: Iterator should yield all elements based on name/docs.

Classification: Genuine bug if documented to iterate fully.

Hash/Equality Inconsistency

Symptom: Equal objects have different hashes.

@given(valid_objects())
def test_hash_equality(obj):
    obj2 = create_equal_copy(obj)
    assert obj == obj2
    assert hash(obj) == hash(obj2)  # Fails

Grounding check: Python requires a == b implies hash(a) == hash(b).

Classification: Genuine bug - violates language contract.

Roundtrip Failure on Edge Cases

Symptom: Encode/decode doesn't preserve input.

@given(st.text())
def test_roundtrip(s):
    assert decode(encode(s)) == s  # Fails for s='\\uD800'

Grounding check: Is '\\uD800' (lone surrogate) valid input?

Classification:

• If docs say "valid UTF-8 only" → Strategy bug, fix filter
• If docs say "any string" → Genuine bug, report it

Format String Errors

Symptom: String formatting crashes on certain inputs.

@given(st.text())
def test_format_safe(template):
    format_message(template)  # Raises on '{unclosed'

Grounding check: Does function claim to handle arbitrary strings?

Classification:

• If user-facing, should handle gracefully → Genuine bug
• If internal API with preconditions → Check preconditions met

When NOT to Report

Do not report as bugs:

1. Precondition violations: If docs say "positive integers only" and you passed -1
2. Undefined behavior: Spec explicitly says behavior is undefined
3. Implementation details: Relying on undocumented internal behavior
4. Platform-specific: Bug only on unusual platform/version
5. Test artifact: Failure disappears with realistic constraints

Confidence Threshold

Report only when you can answer YES to all:

1. Did you reproduce with a minimal example?
2. Did you verify the property against docs/types/docstrings?
3. Can you point to a specific documented guarantee that's violated?
4. Is the failing input within the documented domain?
5. Have you ruled out test bugs and ambiguous specs?

If uncertain on any point, open a discussion first, not a bug report.