Iterative Optimization Loop Skill — Requirements Brainstorm

Problem Statement

CE has strong knowledge-compounding (learn from past work) and multi-agent review (quality gates), but no skill for metric-driven iterative optimization — the pattern where you define a measurable goal, build measurement scaffolding, then run an automated loop that tries many approaches, measures each, keeps improvements, and converges toward the best solution.

Motivating Example

A project builds issue/PR clusters for a large open-source repo. Currently only ~20% of issues/PRs land in clusters with >1 item. The suspected achievable target is ~95%. Getting there requires testing many hypotheses:

• Extracting signal (unique user-entered text) from noise (PR/issue template boilerplate that makes all vectors too similar)
• Using issue-to-PR links as a new clustering signal
• Adjusting similarity thresholds
• Trying different embedding models or chunking strategies
• Combining multiple signals (text similarity + link graph + label overlap + author patterns)
• Pre-filtering or normalizing template sections before embedding

No single hypothesis will get from 20% to 95%. It requires systematic experimentation — trying dozens or hundreds of variations, measuring each, and building on successes.

Landscape Analysis

Karpathy's AutoResearch (March 2026, 21k+ stars)

The simplest and most influential model. Core design:

• One mutable file (train.py) — the agent edits only this
• One immutable evaluator (prepare.py) — the agent cannot touch measurement
• One instruction file (program.md) — defines objectives, constraints, stopping criteria
• One metric (val_bpb) — scalar, lower is better
• Linear keep/revert loop: modify -> commit -> run -> measure -> if improved keep, else git reset
• History: results.tsv accumulates all experiment results; git log preserves successful commits
• Result: 700 experiments in 2 days, 20 discovered optimizations, ~12 experiments/hour

Strengths: Dead simple. Git-native history. Easy to understand and debug. Weaknesses: Linear — can't explore multiple directions simultaneously. Single scalar metric. No backtracking to earlier promising states.

AIDE / WecoAI

• Tree search in solution space — each script is a node, LLM patches spawn children
• Can backtrack to any previous node and explore alternatives
• 4x more Kaggle medals than linear agents on MLE-Bench
• More complex but better at escaping local optima

Sakana AI Scientist v2

• Agentic tree search with parallel experiment execution
• VLM feedback for analyzing figures
• Full paper generation with automated peer review
• Overkill for code optimization but shows the value of tree-structured exploration

DSPy (Stanford)

• Automated prompt/weight optimization for LLM programs
• Bayesian optimization (MIPROv2), iterative feedback (GEPA), coordinate ascent (COPRO)
• Shows that different optimization strategies suit different problem shapes

Existing Claude Code AutoResearch Forks

• uditgoenka/autoresearch — packages the pattern as a Claude Code skill
• autoexp — generalized for any project with a quantifiable metric
• Multiple teams report 50-80% improvements over 30-70 iterations overnight

Key Design Decisions

1. Linear vs. Tree Search

Recommendation: Start with linear keep/revert (Karpathy model) as the default. Add optional "branch point" support where the user can snapshot the current best and start a new exploration direction. Each direction is its own branch. This keeps the core loop simple while allowing multi-direction exploration when needed.

2. What Gets Measured — The Three-Tier Metric Architecture

AutoResearch uses a single scalar metric (val_bpb). That works when you have an objective function with clear ground truth. Most real-world optimization problems don't — especially when the quality of the output requires human judgment.

Key insight: Hard scalar metrics are often the wrong optimization target. For clustering, "bigger clusters" isn't inherently better. "Fewer singletons" isn't inherently better. A solution with 35% singletons where every cluster is coherent beats a solution with 5% singletons where clusters are garbage. Hard metrics catch degenerate solutions; quality requires judgment.

Three tiers:

1. Degenerate-case gates (hard, cheap, fully automated):

   • Catch obviously broken solutions before expensive evaluation
   • Examples: "all items in 1 cluster" (degenerate merge), "all singletons" (degenerate split), "runtime > 10 minutes" (performance regression)
   • These are fast boolean checks: pass/fail. If any gate fails, the experiment is immediately reverted without running the expensive judge
   • Think of these as "sanity checks" not "optimization targets"

2. LLM-as-judge quality score (the actual optimization target):

   • For problems where quality requires judgment, this IS the primary metric
   • Cost-controlled via stratified sampling (not exhaustive)
   • Produces a scalar score the loop can optimize against
   • Can include multiple dimensions (coherence, granularity, completeness)
   • See detailed design below

3. Diagnostics (logged for understanding, not gated on):

   • Distribution stats, counts, histograms
   • Useful for understanding WHY a judge score changed
   • Examples: median cluster size, singleton %, largest cluster size, cluster count
   • Logged in the experiment record but never used for keep/revert decisions

When to use which configuration:

Recommendation: Support all three tiers. The user declares whether the primary optimization target is a hard metric or an LLM-judge score. Degenerate gates always run first (cheap). Judge runs only on experiments that pass gates.

3. What the Agent Can Edit

AutoResearch constrains the agent to one file. This is elegant but too restrictive for most software projects.

Recommendation: Define an explicit allowlist of mutable files/directories and an explicit denylist (measurement harness, test fixtures, evaluation data). The agent operates within the allowlist. The measurement harness is immutable — the agent cannot game the metric by changing how it's measured.

4. Measurement Scaffolding First

This is critical and distinguishes this from "just run the code in a loop":

1. Define the measurement spec before any optimization begins
2. Build and validate the measurement harness — ensure it produces reliable, reproducible results
3. Establish baseline — run the harness on the current code to get starting metrics
4. Only then begin the optimization loop

Recommendation: Make this a hard phase gate. The skill refuses to enter the optimization loop until the measurement harness passes a validation check (runs successfully, produces expected metric types, baseline is recorded).

5. History and Memory

What gets remembered across iterations:

• Results log: Every experiment's metrics, hypothesis, and outcome (kept/reverted)
• Git history: Successful experiments are commits; branches are preserved
• Hypothesis log: What was tried, why, what was learned — prevents re-trying failed approaches
• Strategy evolution: As the agent learns what works, it should adapt its exploration strategy

Recommendation: A structured experiment log (YAML or JSON) that captures: iteration number, hypothesis, changes made, metrics before/after, outcome (kept/reverted/error), and learnings. The agent reads this before proposing the next hypothesis. Git branches are preserved for all kept experiments.

6. How Long It Runs

• AutoResearch runs "indefinitely until manually stopped"
• Real-world needs: time budgets, iteration budgets, metric targets, or "until no improvement for N iterations"

Recommendation: Support multiple stopping criteria (any can trigger stop):

• Target metric reached
• Max iterations
• Max wall-clock time
• No improvement for N consecutive iterations
• Manual stop (user interrupts)

7. Parallelism

AutoResearch is single-threaded. AIDE and AI Scientist run parallel experiments. For CE:

• Phase 1 (v1): Single-threaded linear loop. Simple, debuggable, works with git worktrees.
• Phase 2 (future): Parallel experiments using multiple worktrees or Codex sandboxes. Each experiment is independent.

Recommendation: Start single-threaded. Design the experiment log and branching model to support parallelism later.

8. Integration with Existing CE Skills

The optimization loop should compose with existing CE capabilities:

• /ce:ideate or /ce:brainstorm to generate initial hypothesis space
• Learnings researcher to check if similar optimization was done before
• /ce:compound to capture the winning strategy as institutional knowledge after the loop completes
• /ce:review optionally on the final winning diff before it's merged

Proposed Skill: /ce-optimize

Workflow Phases

Phase 0: Setup
  |-- Read/create optimization spec (target metric, guard rails, mutable files, constraints)
  |-- Search learnings for prior related optimization attempts
  '-- Validate spec completeness

Phase 1: Measurement Scaffolding (HARD GATE - user must approve before Phase 2)
  |-- If user provides harness:
  |     |-- Review docs (or document usage if undocumented)
  |     |-- Run harness once against current implementation
  |     '-- Confirm baseline measurement is accurate with user
  |-- If agent builds harness:
  |     |-- Build measurement harness (immutable evaluator)
  |     |-- Run validation: harness executes, produces expected metric types
  |     '-- Establish baseline metrics
  |-- Parallelism readiness probe:
  |     |-- Check for hardcoded ports -> parameterize via env var
  |     |-- Check for shared DB files (SQLite, etc.) -> plan copy strategy
  |     |-- Check for shared external services -> warn user
  |     |-- Check for exclusive resource needs (GPU, etc.)
  |     '-- Produce parallel_readiness assessment
  |-- Stability validation (if mode: repeat):
  |     |-- Run harness repeat_count times
  |     |-- Verify variance is within noise_threshold
  |     '-- Confirm aggregation method produces stable baseline
  '-- GATE: Present baseline + parallel readiness to user. Refuse to proceed until approved.

Phase 2: Hypothesis Generation + Dependency Approval
  |-- Analyze the problem space (read code, understand current approach)
  |-- Generate initial hypothesis list (agent + optionally /ce:ideate)
  |-- Prioritize by expected impact and feasibility
  |-- Identify new dependencies across ALL planned hypotheses
  |-- Present dependency list for bulk approval
  '-- Record hypothesis backlog (with dep approval status per hypothesis)

Phase 3: Optimization Loop (repeats in parallel batches)
  |-- Select batch of hypotheses (batch_size = min(backlog, max_concurrent))
  |     '-- Prefer diversity: mix different hypothesis categories per batch
  |-- For each experiment in batch (PARALLEL by default):
  |     |-- Create worktree or Codex sandbox
  |     |-- Copy shared resources (DB files, data files)
  |     |-- Apply parameterization (ports, env vars)
  |     |-- Implement hypothesis (within mutable scope)
  |     |-- Run measurement harness (respecting stability config)
  |     '-- Collect metrics + diff
  |-- Wait for batch completion
  |-- Evaluate results:
  |     |-- Rank by primary metric improvement
  |     |-- Filter by guard rails (reject any that violate)
  |     |-- If best > current: KEEP (merge to optimization branch)
  |     |-- If best has unapproved dep: mark deferred_needs_approval
  |     '-- All others: REVERT (log results, clean up worktrees)
  |-- Handle unapproved deps:
  |     '-- Set aside, don't block pipeline, batch-ask at end or check-in
  |-- Update experiment log with ALL results (kept + reverted)
  |-- Re-baseline: remaining hypotheses evaluated against new best
  |-- Generate new hypotheses based on learnings from this batch
  |-- Check stopping criteria
  '-- Next batch

Phase 4: Wrap-Up
  |-- Present deferred hypotheses needing dep approval (if any)
  |-- Summarize results: baseline -> final metrics, total iterations, kept improvements
  |-- Preserve ALL experiment branches for reference
  |-- Optionally run /ce:review on cumulative diff
  |-- Optionally run /ce:compound to capture winning strategy as learning
  '-- Report to user

Optimization Spec File Format

See "Updated Spec File Format" in the Resolved Design Decisions section below for the full spec with parallel execution and stability config.

Experiment Log Format

# .context/compound-engineering/optimize/experiment-log.yaml
spec: "improve-issue-clustering"

baseline:
  timestamp: "2026-03-29T10:00:00Z"
  gates:
    largest_cluster_pct: 0.02
    singleton_pct: 0.79
    cluster_count: 342
    runtime_seconds: 45
  diagnostics:
    singleton_pct: 0.79
    median_cluster_size: 2
    cluster_count: 342
    avg_cluster_size: 2.8
    p95_cluster_size: 7
  judge:
    mean_score: 3.1
    pct_scoring_4plus: 0.33
    mean_distinct_topics: 1.8
    singleton_false_negative_pct: 0.45   # 45% of sampled singletons should be clustered
    sample_seed: 42
    judge_cost_usd: 0.42

experiments:
  - iteration: 1
    batch: 1
    hypothesis: "Remove PR template boilerplate before embedding to reduce noise"
    category: "signal-extraction"
    changes:
      - file: "src/preprocessing/text_cleaner.py"
        summary: "Added template detection and removal using common PR template patterns"
    gates:
      largest_cluster_pct: 0.03
      singleton_pct: 0.62
      cluster_count: 489
      runtime_seconds: 48
    gates_passed: true
    diagnostics:
      singleton_pct: 0.62
      median_cluster_size: 3
      cluster_count: 489
      avg_cluster_size: 3.4
    judge:
      mean_score: 3.8
      pct_scoring_4plus: 0.57
      mean_distinct_topics: 1.4
      singleton_false_negative_pct: 0.31
      judge_cost_usd: 0.38
    outcome: "kept"
    primary_delta: "+0.7"       # mean_score: 3.1 -> 3.8
    learnings: "Template removal signifi