Mock CLI: Cost-Free E2E Testing

Problem Statement

Running E2E tests against real AI backends (Claude, Kiro, Gemini, etc.) has several problems:

1. Cost: Each test run consumes API credits
2. Speed: Network latency and API rate limits slow down CI/CD
3. Reliability: Network issues and API availability affect test stability
4. Determinism: AI responses vary, making tests non-deterministic

Teams need a way to run E2E tests that:

• Costs nothing (no API calls)
• Runs fast (no network latency)
• Is deterministic (same output every time)
• Validates the full orchestration loop (not just unit tests)

Solution Overview

The mock-cli subcommand replays pre-recorded JSONL cassettes instead of invoking real AI backends. This enables:

• Zero-cost testing: No API calls, no credits consumed
• Fast execution: Instant or accelerated replay (10x+ speed)
• Deterministic output: Same cassette = same output every time
• Full integration: Tests the complete orchestration loop via PTY

The mock CLI acts as a drop-in replacement for real backends by implementing the same command-line interface that Ralph expects.

How It Works

Architecture

ralph-e2e --mock
    │
    ├─ Writes ralph.yml with custom backend
    │  cli:
    │    backend: custom
    │    command: ralph-e2e
    │    args: ["mock-cli", "--cassette", "path/to/cassette.jsonl"]
    │
    └─ ralph run (orchestrator)
        │
        └─ Spawns: ralph-e2e mock-cli --cassette cassettes/e2e/connect.jsonl
            │
            ├─ SessionPlayer: Reads JSONL cassette
            │   └─ Extracts ux.terminal.write events
            │
            ├─ Replays output to stdout (via PTY)
            │
            └─ WhitelistExecutor: Runs approved local commands
                └─ ralph task add, ralph tools memory add, etc.

Cassette Format

Cassettes are JSONL files containing timestamped events from the SessionRecorder:

{"ts":1000,"event":"ux.terminal.write","data":{"bytes":"UE9ORw==","stdout":true,"offset_ms":0}}
{"ts":1100,"event":"bus.publish","data":{"command":"ralph task add 'test'"}}
{"ts":1200,"event":"ux.terminal.write","data":{"bytes":"RG9uZQ==","stdout":true,"offset_ms":200}}

Each line is a JSON object with:

• ts: Unix timestamp in milliseconds
• event: Event type (e.g., ux.terminal.write, bus.publish)
• data: Event-specific payload

The mock CLI extracts:

1. Terminal writes (ux.terminal.write) → replayed to stdout
2. Commands (bus.publish with command field) → executed if whitelisted

Cassette Naming Convention

Cassettes are stored in cassettes/e2e/ with the following resolution order:

1. Backend-specific: <scenario-id>-<backend>.jsonl

   • Example: connect-claude.jsonl, task-add-kiro.jsonl
   • Used when backend-specific behavior differs

2. Generic fallback: <scenario-id>.jsonl

   • Example: connect.jsonl, task-add.jsonl
   • Used when behavior is identical across backends

If neither exists, the test fails fast with a clear error.

Usage Guide

Running E2E Tests in Mock Mode

# Run all E2E tests with mock backends (zero cost)
ralph-e2e --mock

# Run with accelerated replay (10x speed)
ralph-e2e --mock --mock-speed 10.0

# Run with instant replay (no delays)
ralph-e2e --mock --mock-speed 0.0

# Run specific scenarios
ralph-e2e --mock --filter connect

# Custom cassette directory (default: cassettes/e2e)
ralph-e2e --mock --cassette-dir /path/to/cassettes

Direct Mock CLI Invocation

The mock CLI is typically invoked by Ralph as a custom backend, but you can run it directly for testing:

# Basic replay
ralph-e2e mock-cli --cassette cassettes/e2e/connect.jsonl

# With speed adjustment (10x faster)
ralph-e2e mock-cli --cassette cassettes/e2e/connect.jsonl --speed 10.0

# With command execution whitelist
ralph-e2e mock-cli \\
  --cassette cassettes/e2e/task-add.jsonl \\
  --allow "ralph task add,ralph tools memory add"

# Check version (for backend availability checks)
ralph-e2e mock-cli --version

Prerequisites

1. Cassette files: Must exist in cassettes/e2e/ directory
2. Ralph installed: Required for whitelisted command execution
3. Workspace setup: Mock CLI runs in the scenario workspace directory

Recording New Cassettes

To create cassettes for new scenarios:

# Run E2E test with real backend and session recording
ralph run --record-session cassettes/e2e/my-scenario-claude.jsonl

# Or use the E2E harness with recording enabled
# (Implementation detail: E2E harness should support --record flag)

API Reference

Command-Line Interface

ralph-e2e mock-cli [OPTIONS]

OPTIONS:
    --cassette        Path to JSONL cassette file (required)
    --speed          Replay speed multiplier (default: 0.0 = instant)
                           1.0 = real-time, 10.0 = 10x faster
    --allow            Comma-separated command prefixes to whitelist
                           Example: "ralph task add,ralph tools memory add"
    --version              Print version and exit (for availability checks)
    -h, --help             Print help information

Exit Codes

Environment Variables

Cassette Resolution API

The CassetteResolver provides programmatic access to cassette resolution:

use ralph_e2e::mock::{CassetteResolver, MockConfig};
use ralph_e2e::Backend;

// Create resolver
let resolver = CassetteResolver::new("cassettes/e2e");

// Resolve cassette for scenario + backend
let path = resolver.resolve("connect", Backend::Claude)?;
// Returns: cassettes/e2e/connect-claude.jsonl (or connect.jsonl fallback)

// Get all candidate paths (for debugging)
let candidates = resolver.candidates("connect", Backend::Claude);
// Returns: ["cassettes/e2e/connect-claude.jsonl", "cassettes/e2e/connect.jsonl"]

Mock Configuration API

use ralph_e2e::mock::MockConfig;

// Default config (instant replay, standard whitelist)
let config = MockConfig::default();

// Custom config
let config = MockConfig::new("/custom/cassettes")
    .with_speed(10.0)
    .with_allow_commands("ralph task add,ralph task close");

// Disable command execution
let config = MockConfig::default().without_commands();

Edge Cases and Limitations

What Happens When...

Cassette is missing?

Behavior: Test fails immediately with clear error message

Error: cassette not found for scenario 'connect' backend 'claude'
Tried:
  - cassettes/e2e/connect-claude.jsonl
  - cassettes/e2e/connect.jsonl

Solution: Record a cassette for this scenario or use a generic fallback

Cassette contains invalid JSONL?

Behavior: Parse error with line number and details

Error: cassette parse error in cassettes/e2e/connect.jsonl
Line 5: expected value at line 1 column 1

Solution: Validate cassette format or re-record

Command is not whitelisted?

Behavior: Command is skipped with warning to stderr

[mock-cli] Skipping non-whitelisted command: rm -rf /

Solution: Add command to whitelist if it's safe and necessary

Whitelisted command fails?

Behavior: Warning logged, replay continues (non-fatal)

[mock-cli] Warning: command 'ralph task close invalid-id' exited with status 1

Rationale: Command failures during replay shouldn't break the test unless the scenario explicitly checks for them

Cassette contains no terminal writes?

Behavior: Mock CLI outputs nothing, exits successfully

Use case: Scenarios that only test side effects (tasks, memories) without output validation

Speed is negative?

Behavior: Clamped to 0.0 (instant replay)

let config = MockConfig::default().with_speed(-5.0);
assert_eq!(config.speed, 0.0);

Multiple backends use same cassette?

Behavior: Generic cassette (<scenario>.jsonl) is used for all backends

Use case: Scenarios where backend behavior is identical (e.g., connectivity checks)

Limitations

1. No shell features: Command whitelist does NOT support pipes, redirects, or variable expansion

   • ✅ Allowed: ralph task add 'test'
   • ❌ Not allowed: ralph task add 'test' | grep foo

2. No network access: Mock CLI cannot make real API calls or network requests

   • Use real backend mode for integration tests requiring network

3. Timing approximation: Replay timing is approximate, not exact

   • Sufficient for E2E validation, not for performance benchmarking

4. Command execution is synchronous: Whitelisted commands run sequentially

   • No parallel execution or background processes

5. PTY limitations: Mock CLI outputs to PTY, which may affect ANSI escape sequences

   • Most terminal output works correctly, but complex TUI interactions may differ

Examples

Example 1: Basic Connectivity Test

Scenario: Verify Ralph can connect to backend and receive output

Cassette (cassettes/e2e/connect.jsonl):

{"ts":1000,"event":"ux.terminal.write","data":{"bytes":"UE9ORw==","stdout":true,"offset_ms":0}}

Usage:

ralph-e2e --mock --filter connect

Expected: Test passes, output contains "PONG"

Example 2: Task Creation with Side Effects

Scenario: Verify Ralph can create tasks via ralph task add

Cassette (cassettes/e2e/task-add.jsonl):

{"ts":1000,"event":"ux.terminal.write","data":{"bytes":"Q3JlYXRpbmcgdGFzaw==","stdout":true,"offset_ms":0}}
{"ts":1100,"event":"bus.publish","data":{"command":"ralph task add 'test task' -p 1"}}
{"ts":1200,"event":"ux.terminal.write","data":{"bytes":"VGFzayBjcmVhdGVk","stdout":true,"offset_ms":100}}

Usage:

ralph-e2e mock-cli \\
  --cassette cassettes/e2e/task-add.jsonl \\
  --allow "ralph task add"

Expected:

• Output contains "Creating task" and "Task created"
• .agent/tasks.jsonl contains new task entry

Example 3: Accelerated Replay for CI

Scenario: Run full E2E suite quickly in CI pipeline

Usage:

# Run all tests with 10x speed (no delays)
ralph-e2e --mock --mock-speed 0.0

Expected: All tests complete in seconds instead of minutes

Example 4: Backend-Specific Behavior

Scenario: Test Claude-specific output format

Cassettes:

• cassettes/e2e/format-claude.jsonl (Claude-specific)
• cassettes/e2e/format-kiro.jsonl (Kiro-specific)
• cassettes/e2e/format.jsonl (generic fallback)

Usage:

# Runs with backend-specific cassettes
ralph-e2e --mock --filter format

Expected: Each backend uses its specific cassette, falls back to generic if missing

Example 5: Error Scenario Testing

Scenario: Verify Ralph handles backend timeout gracefully

Cassette (cassettes/e2e/timeout-handling.jsonl):

{"ts":1000,"event":"ux.terminal.write","data":{"bytes":"U3RhcnRpbmc=","stdout":true,"offset_ms":0}}
{"ts":31000,"event":"ux.terminal.write","data":{"bytes":"VGltZW91dA==","stdout":true,"offset_ms":30000}}

Usage:

ralph-e2e --mock --filter timeout-handling --mock-speed 10.0

Expected: Test validates timeout handling (3 seconds at 10x speed)

Troubleshooting

Problem: "cassette not found" error

Symptoms:

Error: cassette not found for scenario 'my-test' backend 'claude'

Solutions:

1. Check cassette file exists: ls cassettes/e2e/my-test*.jsonl
2. Verify naming convention: <scenario-id>-<backend>.jsonl or <scenario-id>.jsonl
3. Record a new cassette with real backend
4. Use generic cassette (remove backend suffix)

Problem: Cassette parse error

Symptoms:

Error: cassette parse error in cassettes/e2e/test.jsonl

Solutions:

1. Validate JSONL format: jq . cassettes/e2e/test.jsonl
2. Check for trailing commas or invalid JSON
3. Re-record cassette from scratch

Problem: Commands not executing

Symptoms: Expected side effects (tasks, memories) not present

Solutions:

1. Verify whitelist includes command: --allow "ralph task add"
2. Check cassette contains bus.publish events with commands
3. Ensure commands are in correct format (no shell features)
4. Run with verbose logging to see skipped commands

Problem: Output differs from real backend

Symptoms: Mock output doesn't match real backend behavior

Solutions:

1. Re-record cassette with latest backend version
2. Check for backend-specific cassette: <scenario>-<backend>.jsonl
3. Verify cassette was recorded in same environment (PTY vs non-PTY)

Problem: Tests pass in mock mode but fail with real backend

Symptoms: Mock tests pass, real E2E tests fail

Root cause: Cassette is outdated or doesn't reflect real behavior

Solutions:

1. Re-record cassettes with current backend
2. Run real E2E tests periodically (e.g., nightly)
3. Use mock mode for fast feedback, real mode for validation

Best Practices

When to Use Mock Mode

✅ Use mock mode for:

• Fast feedback during development
• CI/CD pipelines (cost and speed)
• Regression testing (deterministic output)
• Testing error scenarios (timeouts, failures)

❌ Don't use mock mode for:

• Validating new backend integrations
• Testing actual AI behavior changes
• Performance benchmarking
• Network-dependent scenarios

Cassette Management

1. Version control: Commit cassettes to git for reproducibility
2. Naming convention: Use descriptive scenario IDs
3. Backend-specific: Only create when behavior differs
4. Regular updates: Re-record when backend behavior changes
5. Minimal cassettes: Keep cassettes small and focused

Whitelist Safety

1. Principle of least privilege: Only whitelist necessary commands
2. No destructive commands: Never whitelist rm, mv, etc.
3. Prefix matching: Use specific prefixes (ralph task add, not ralph)
4. Review regularly: Audit whitelist for unnecessary entries

Testing Strategy

1. Mock for speed: Run mock tests on every commit
2. Real for validation: Run real E2E tests nightly or weekly
3. Hybrid approach: Mock for most scenarios, real for critical paths
4. Cassette freshness: Re-record cassettes quarterly or when backends update