E2E Test Cassettes

This directory contains recorded cassettes for mock-mode E2E testing. Cassettes are JSONL files that record the output from real Ralph sessions, enabling deterministic, cost-free test execution.

Directory Structure

cassettes/e2e/
├── README.md           # This file
├── connect.jsonl       # Generic connectivity cassette (all backends)
├── events.jsonl        # Event XML parsing test
├── completion.jsonl    # LOOP_COMPLETE detection test
├── single-iter.jsonl   # Single iteration orchestration
├── multi-iter.jsonl    # Multi-iteration orchestration
└── ...

Available Cassettes

Known Limitations

1. Multi-iteration scenarios: Mock-cli replays entire cassette in one invocation, so Ralph sees only one iteration
2. File write assertions: Scenarios checking scratchpad/artifact content fail unless whitelisted commands execute
3. Task/Memory scenarios: Require cassettes with bus.publish events containing whitelisted commands

Naming Convention

Cassettes follow this naming pattern:

• <scenario-id>.jsonl - Generic fallback cassette
• <scenario-id>-<backend>.jsonl - Backend-specific cassette

When running in mock mode, the cassette resolver checks for:

1. <scenario>-<backend>.jsonl (backend-specific, preferred)
2. <scenario>.jsonl (generic fallback)

Recording New Cassettes

To record a new cassette from a live session:

# Record with ralph's built-in recording
cargo run --bin ralph -- run \\
  -c ralph.yml \\
  --record-session cassettes/e2e/my-scenario.jsonl \\
  -p "Your prompt here"

Cassette Format

Each line is a JSON object with these fields:

• ts: Unix timestamp in milliseconds
• event: Event type (e.g., ux.terminal.write, bus.publish, _meta.iteration)
• data: Event-specific data

Event Types

Terminal Write Data

{
  "bytes": "SGVsbG8=",  // Base64-encoded output
  "stdout": true,       // true for stdout, false for stderr
  "offset_ms": 100      // Time offset from session start
}

Usage

Run E2E tests in mock mode:

# Run all tests with cassettes
cargo run -p ralph-e2e -- --mock

# Run with specific cassette directory
cargo run -p ralph-e2e -- --mock --cassette-dir ./my-cassettes

# Run with real-time playback (1x speed)
cargo run -p ralph-e2e -- --mock --mock-speed 1.0

# Check cassette availability
cargo run -p ralph-e2e -- --mock --list

Creating Cassettes for New Scenarios

1. Run the scenario against a live backend with recording enabled
2. Copy the recorded JSONL to the appropriate location
3. Name it according to the convention above
4. Run ralph-e2e --mock --filter <scenario-id> to verify