---
title: "Iterative Optimization Loop Skill — Requirements Brainstorm"
description: "CE has strong knowledge-compounding (learn from past work) and multi-agent review (quality gates), but no skill for **metric-driven iterative optimization** — the pattern where you define a measurable goal, build measurement scaffolding, then run an automated loop that tries many approaches, measures each, keeps improvements, and converges toward the best solution."
type: skill
canonical_url: https://claudary.paisolsolutions.com/skills/2026-03-29-iterative-optimization-loop-requirements
source: "Claudary"
difficulty: intermediate
author: "Claude Code Knowledge Pack"
date: 2026-07-10T10:59:03.529Z
license: CC-BY-4.0
attribution: "Iterative Optimization Loop Skill — Requirements Brainstorm — Claudary (https://claudary.paisolsolutions.com/skills/2026-03-29-iterative-optimization-loop-requirements)"
---

# Iterative Optimization Loop Skill — Requirements Brainstorm
CE has strong knowledge-compounding (learn from past work) and multi-agent review (quality gates), but no skill for **metric-driven iterative optimization** — the pattern where you define a measurable goal, build measurement scaffolding, then run an automated loop that tries many approaches, measures each, keeps improvements, and converges toward the best solution.

## Overview

# Iterative Optimization Loop Skill — Requirements Brainstorm

## Problem Statement

CE has strong knowledge-compounding (learn from past work) and multi-agent review (quality gates), but no skill for **metric-driven iterative optimization** — the pattern where you define a measurable goal, build measurement scaffolding, then run an automated loop that tries many approaches, measures each, keeps improvements, and converges toward the best solution.

### Motivating Example

A project builds issue/PR clusters for a large open-source repo. Currently only ~20% of issues/PRs land in clusters with >1 item. The suspected achievable target is ~95%. Getting there requires testing many hypotheses:

- Extracting signal (unique user-entered text) from noise (PR/issue template boilerplate that makes all vectors too similar)
- Using issue-to-PR links as a new clustering signal
- Adjusting similarity thresholds
- Trying different embedding models or chunking strategies
- Combining multiple signals (text similarity + link graph + label overlap + author patterns)
- Pre-filtering or normalizing template sections before embedding

No single hypothesis will get from 20% to 95%. It requires systematic experimentation — trying dozens or hundreds of variations, measuring each, and building on successes.

## Landscape Analysis

### Karpathy's AutoResearch (March 2026, 21k+ stars)

The simplest and most influential model. Core design:

- **One mutable file** (`train.py`) — the agent edits only this
- **One immutable evaluator** (`prepare.py`) — the agent cannot touch measurement
- **One instruction file** (`program.md`) — defines objectives, constraints, stopping criteria
- **One metric** (`val_bpb`) — scalar, lower is better
- **Linear keep/revert loop**: modify -> commit -> run -> measure -> if improved keep, else `git reset`
- **History**: `results.tsv` accumulates all experiment results; git log preserves successful commits
- **Result**: 700 experiments in 2 days, 20 discovered optimizations, ~12 experiments/hour

**Strengths**: Dead simple. Git-native history. Easy to understand and debug.
**Weaknesses**: Linear — can't explore multiple directions simultaneously. Single scalar metric. No backtracking to earlier promising states.

### AIDE / WecoAI

- **Tree search** in solution space — each script is a node, LLM patches spawn children
- Can backtrack to any previous node and explore alternatives
- 4x more Kaggle medals than linear agents on MLE-Bench
- More complex but better at escaping local optima

### Sakana AI Scientist v2

- **Agentic tree search** with parallel experiment execution
- VLM feedback for analyzing figures
- Full paper generation with automated peer review
- Overkill for code optimization but shows the value of tree-structured exploration

### DSPy (Stanford)

- Automated prompt/weight optimization for LLM programs
- Bayesian optimization (MIPROv2), iterative feedback (GEPA), coordinate ascent (COPRO)
- Shows that different optimization strategies suit different problem shapes

### Existing Claude Code AutoResearch Forks

- `uditgoenka/autoresearch` — packages the pattern as a Claude Code skill
- `autoexp` — generalized for any project with a quantifiable metric
- Multiple teams report 50-80% improvements over 30-70 iterations overnight

## Key Design Decisions

### 1. Linear vs. Tree Search

| Approach | Pros | Cons |
|---|---|---|
| Linear (autoresearch) | Simple, easy to understand, git-native | Can't explore multiple directions, stuck in local optima |
| Tree search (AIDE) | Can backtrack, explore alternatives | More complex state management, harder to review |
| Hybrid: linear with manual branch points | Best of both — simple default, user chooses when to fork | Requires user interaction to fork |

**Recommendation**: Start with linear keep/revert (Karpathy model) as the default. Add optional "branch point" support where the user can snapshot the current best and start a new exploration direction. Each direction is its own branch. This keeps the core loop simple while allowing multi-direction exploration when needed.

### 2. What Gets Measured — The Three-Tier Metric Architecture

AutoResearch uses a single scalar metric (val_bpb). That works when you have an objective function with clear ground truth. Most real-world optimization problems don't — especially when the quality of the output requires human judgment.

**Key insight**: Hard scalar metrics are often the wrong optimization target. For clustering, "bigger clusters" isn't inherently better. "Fewer singletons" isn't inherently better. A solution with 35% singletons where every cluster is coherent beats a solution with 5% singletons where clusters are garbage. Hard metrics catch *degenerate* solutions; *quality* requires judgment.

**Three tiers**:

1. **Degenerate-case gates** (hard, cheap, fully automated):
   - Catch obviously broken solutions before expensive evaluation
   - Examples: "all items in 1 cluster" (degenerate merge), "all singletons" (degenerate split), "runtime > 10 minutes" (performance regression)
   - These are fast boolean checks: pass/fail. If any gate fails, the experiment is immediately reverted without running the expensive judge
   - Think of these as "sanity checks" not "optimization targets"

2. **LLM-as-judge quality score** (the actual optimization target):
   - For problems where quality requires judgment, this IS the primary metric
   - Cost-controlled via stratified sampling (not exhaustive)
   - Produces a scalar score the loop can optimize against
   - Can include multiple dimensions (coherence, granularity, completeness)
   - See detailed design below

3. **Diagnostics** (logged for understanding, not gated on):
   - Distribution stats, counts, histograms
   - Useful for understanding WHY a judge score changed
   - Examples: median cluster size, singleton %, largest cluster size, cluster count
   - Logged in the experiment record but never used for keep/revert decisions

**When to use which configuration**:

| Problem Type | Degenerate Gates | Primary Metric | Example |
|---|---|---|---|
| Objective function exists | Yes | Hard metric (scalar) | Build time, test pass rate, API latency |
| Quality requires judgment | Yes | LLM-as-judge score | Clustering quality, search relevance, content generation |
| Hybrid | Yes | Hard metric + LLM-judge as guard rail | Latency (optimize) + response quality (must not drop) |

**Recommendation**: Support all three tiers. The user declares whether the primary optimization target is a hard metric or an LLM-judge score. Degenerate gates always run first (cheap). Judge runs only on experiments that pass gates.

### 3. What the Agent Can Edit

AutoResearch constrains the agent to one file. This is elegant but too restrictive for most software projects.

**Recommendation**: Define an explicit allowlist of mutable files/directories and an explicit denylist (measurement harness, test fixtures, evaluation data). The agent operates within the allowlist. The measurement harness is immutable — the agent cannot game the metric by changing how it's measured.

### 4. Measurement Scaffolding First

This is critical and distinguishes this from "just run the code in a loop":

1. **Define the measurement spec** before any optimization begins
2. **Build and validate the measurement harness** — ensure it produces reliable, reproducible results
3. **Establish baseline** — run the harness on the current code to get starting metrics
4. Only then begin the optimization loop

**Recommendation**: Make this a hard phase gate. The skill refuses to enter the optimization loop until the measurement harness passes a validation check (runs successfully, produces expected metric types, baseline is recorded).

### 5. History and Memory

What gets remembered across iterations:

- **Results log**: Every experiment's metrics, hypothesis, and outcome (kept/reverted)
- **Git history**: Successful experiments are commits; branches are preserved
- **Hypothesis log**: What was tried, why, what was learned — prevents re-trying failed approaches
- **Strategy evolution**: As the agent learns what works, it should adapt its exploration strategy

**Recommendation**: A structured experiment log (YAML or JSON) that captures: iteration number, hypothesis, changes made, metrics before/after, outcome (kept/reverted/error), and learnings. The agent reads this before proposing the next hypothesis. Git branches are preserved for all kept experiments.

### 6. How Long It Runs

- AutoResearch runs "indefinitely until manually stopped"
- Real-world needs: time budgets, iteration budgets, metric targets, or "until no improvement for N iterations"

**Recommendation**: Support multiple stopping criteria (any can trigger stop):
- Target metric reached
- Max iterations
- Max wall-clock time
- No improvement for N consecutive iterations
- Manual stop (user interrupts)

### 7. Parallelism

AutoResearch is single-threaded. AIDE and AI Scientist run parallel experiments. For CE:

- **Phase 1 (v1)**: Single-threaded linear loop. Simple, debuggable, works with git worktrees.
- **Phase 2 (future)**: Parallel experiments using multiple worktrees or Codex sandboxes. Each experiment is independent.

**Recommendation**: Start single-threaded. Design the experiment log and branching model to support parallelism later.

### 8. Integration with Existing CE Skills

The optimization loop should compose with existing CE capabilities:

- **`/ce:ideate`** or **`/ce:brainstorm`** to generate initial hypothesis space
- **Learnings researcher** to check if similar optimization was done before
- **`/ce:compound`** to capture the winning strategy as institutional knowledge after the loop completes
- **`/ce:review`** optionally on the final winning diff before it's merged

## Proposed Skill: `/ce-optimize`

### Workflow Phases

```
Phase 0: Setup
  |-- Read/create optimization spec (target metric, guard rails, mutable files, constraints)
  |-- Search learnings for prior related optimization attempts
  '-- Validate spec completeness

Phase 1: Measurement Scaffolding (HARD GATE - user must approve before Phase 2)
  |-- If user provides harness:
  |     |-- Review docs (or document usage if undocumented)
  |     |-- Run harness once against current implementation
  |     '-- Confirm baseline measurement is accurate with user
  |-- If agent builds harness:
  |     |-- Build measurement harness (immutable evaluator)
  |     |-- Run validation: harness executes, produces expected metric types
  |     '-- Establish baseline metrics
  |-- Parallelism readiness probe:
  |     |-- Check for hardcoded ports -> parameterize via env var
  |     |-- Check for shared DB files (SQLite, etc.) -> plan copy strategy
  |     |-- Check for shared external services -> warn user
  |     |-- Check for exclusive resource needs (GPU, etc.)
  |     '-- Produce parallel_readiness assessment
  |-- Stability validation (if mode: repeat):
  |     |-- Run harness repeat_count times
  |     |-- Verify variance is within noise_threshold
  |     '-- Confirm aggregation method produces stable baseline
  '-- GATE: Present baseline + parallel readiness to user. Refuse to proceed until approved.

Phase 2: Hypothesis Generation + Dependency Approval
  |-- Analyze the problem space (read code, understand current approach)
  |-- Generate initial hypothesis list (agent + optionally /ce:ideate)
  |-- Prioritize by expected impact and feasibility
  |-- Identify new dependencies across ALL planned hypotheses
  |-- Present dependency list for bulk approval
  '-- Record hypothesis backlog (with dep approval status per hypothesis)

Phase 3: Optimization Loop (repeats in parallel batches)
  |-- Select batch of hypotheses (batch_size = min(backlog, max_concurrent))
  |     '-- Prefer diversity: mix different hypothesis categories per batch
  |-- For each experiment in batch (PARALLEL by default):
  |     |-- Create worktree or Codex sandbox
  |     |-- Copy shared resources (DB files, data files)
  |     |-- Apply parameterization (ports, env vars)
  |     |-- Implement hypothesis (within mutable scope)
  |     |-- Run measurement harness (respecting stability config)
  |     '-- Collect metrics + diff
  |-- Wait for batch completion
  |-- Evaluate results:
  |     |-- Rank by primary metric improvement
  |     |-- Filter by guard rails (reject any that violate)
  |     |-- If best > current: KEEP (merge to optimization branch)
  |     |-- If best has unapproved dep: mark deferred_needs_approval
  |     '-- All others: REVERT (log results, clean up worktrees)
  |-- Handle unapproved deps:
  |     '-- Set aside, don't block pipeline, batch-ask at end or check-in
  |-- Update experiment log with ALL results (kept + reverted)
  |-- Re-baseline: remaining hypotheses evaluated against new best
  |-- Generate new hypotheses based on learnings from this batch
  |-- Check stopping criteria
  '-- Next batch

Phase 4: Wrap-Up
  |-- Present deferred hypotheses needing dep approval (if any)
  |-- Summarize results: baseline -> final metrics, total iterations, kept improvements
  |-- Preserve ALL experiment branches for reference
  |-- Optionally run /ce:review on cumulative diff
  |-- Optionally run /ce:compound to capture winning strategy as learning
  '-- Report to user
```

### Optimization Spec File Format

See "Updated Spec File Format" in the Resolved Design Decisions section below for the full spec with parallel execution and stability config.

### Experiment Log Format

```yaml
# .context/compound-engineering/optimize/experiment-log.yaml
spec: "improve-issue-clustering"

baseline:
  timestamp: "2026-03-29T10:00:00Z"
  gates:
    largest_cluster_pct: 0.02
    singleton_pct: 0.79
    cluster_count: 342
    runtime_seconds: 45
  diagnostics:
    singleton_pct: 0.79
    median_cluster_size: 2
    cluster_count: 342
    avg_cluster_size: 2.8
    p95_cluster_size: 7
  judge:
    mean_score: 3.1
    pct_scoring_4plus: 0.33
    mean_distinct_topics: 1.8
    singleton_false_negative_pct: 0.45   # 45% of sampled singletons should be clustered
    sample_seed: 42
    judge_cost_usd: 0.42

experiments:
  - iteration: 1
    batch: 1
    hypothesis: "Remove PR template boilerplate before embedding to reduce noise"
    category: "signal-extraction"
    changes:
      - file: "src/preprocessing/text_cleaner.py"
        summary: "Added template detection and removal using common PR template patterns"
    gates:
      largest_cluster_pct: 0.03
      singleton_pct: 0.62
      cluster_count: 489
      runtime_seconds: 48
    gates_passed: true
    diagnostics:
      singleton_pct: 0.62
      median_cluster_size: 3
      cluster_count: 489
      avg_cluster_size: 3.4
    judge:
      mean_score: 3.8
      pct_scoring_4plus: 0.57
      mean_distinct_topics: 1.4
      singleton_false_negative_pct: 0.31
      judge_cost_usd: 0.38
    outcome: "kept"
    primary_delta: "+0.7"       # mean_score: 3.1 -> 3.8
    learnings: "Template removal signifi

---

Source: [Claudary](https://claudary.paisolsolutions.com/skills/2026-03-29-iterative-optimization-loop-requirements) · https://claudary.paisolsolutions.com