--- title: "feat(ce-optimize): Add iterative optimization loop skill" description: "Add a new `/ce-optimize` skill that implements metric-driven iterative optimization — the pattern where you define a measurable goal, build measurement scaffolding first, then run an automated loop that tries many parallel experiments, measures each against hard gates and/or LLM-as-judge quality scores, keeps improvements, and converges toward the best solution. Inspired by Karpathy's autoresearch" type: skill canonical_url: https://claudary.paisolsolutions.com/skills/2026-03-29-001-feat-iterative-optimization-loop-skill-beta-plan source: "Claudary" difficulty: intermediate author: "Claude Code Knowledge Pack" date: 2026-07-10T10:59:03.529Z license: CC-BY-4.0 attribution: "feat(ce-optimize): Add iterative optimization loop skill — Claudary (https://claudary.paisolsolutions.com/skills/2026-03-29-001-feat-iterative-optimization-loop-skill-beta-plan)" --- # feat(ce-optimize): Add iterative optimization loop skill Add a new `/ce-optimize` skill that implements metric-driven iterative optimization — the pattern where you define a measurable goal, build measurement scaffolding first, then run an automated loop that tries many parallel experiments, measures each against hard gates and/or LLM-as-judge quality scores, keeps improvements, and converges toward the best solution. Inspired by Karpathy's autoresearch ## Overview --- title: "feat(ce-optimize): Add iterative optimization loop skill" type: feat status: completed date: 2026-03-29 origin: docs/brainstorms/2026-03-29-iterative-optimization-loop-requirements.md deepened: 2026-03-29 --- # feat(ce-optimize): Add iterative optimization loop skill ## Overview Add a new `/ce-optimize` skill that implements metric-driven iterative optimization — the pattern where you define a measurable goal, build measurement scaffolding first, then run an automated loop that tries many parallel experiments, measures each against hard gates and/or LLM-as-judge quality scores, keeps improvements, and converges toward the best solution. Inspired by Karpathy's autoresearch but generalized for multi-file code changes, complex metrics, and non-ML domains. ## Problem Frame CE has knowledge-compounding and quality gates but no skill for systematic experimentation. When a developer needs to improve a measurable outcome (clustering quality, build performance, search relevance), they currently iterate manually — one change at a time, eyeballing results. This skill automates the modify-measure-decide cycle, runs experiments in parallel via worktrees or Codex sandboxes, and preserves all experiment history in git for later reference. (see origin: `docs/brainstorms/2026-03-29-iterative-optimization-loop-requirements.md`) ## Requirements Trace - R1. User can define an optimization target (spec file) in <15 minutes - R2. Measurement scaffolding is validated before the loop starts (hard phase gate) - R3. Three-tier metric architecture: degenerate gates (cheap boolean checks) -> LLM-as-judge quality score (sampled, cost-controlled) -> diagnostics (logged, not gated) - R4. LLM-as-judge with stratified sampling and user-defined rubric is a first-class primary metric type, not deferred - R5. Experiments run in parallel by default using worktree isolation or Codex sandboxes - R6. Parallelism blockers (ports, shared DBs, exclusive resources) are actively detected and mitigated during Phase 1 - R7. Dependencies are pre-approved in bulk during hypothesis generation; unapproved deps defer the hypothesis without blocking the pipeline - R8. Flaky metrics are configurable (repeat N times, aggregate via median/mean, noise threshold) - R9. All experiments preserved in git for later reference; experiment log captures hypothesis, metrics, outcome, and learnings - R10. The winning strategy is documented via `/ce:compound` integration - R11. Codex support from v1 using established `codex exec` stdin-pipe pattern - R12. Loop handles failures gracefully (bad experiments don't corrupt state) - R13. Multiple stopping criteria: target reached, max iterations, max hours, plateau (N iterations no improvement), manual stop ## Scope Boundaries - No tree search / backtracking in v1 — linear keep/revert with optional manual branch points only - No batch size adaptation — fixed `max_concurrent`, user-tunable - No LLM-as-judge calibration anchors in v1 — deferred to future iteration - No rubric mid-loop iteration protocol in v1 - No judge cost budget enforcement — cost tracked in log, user decides - This plan covers the skill, reference files, and scripts. It does not cover changes to the CLI converter or other targets ## Context & Research ### Relevant Code and Patterns - **Skill format**: `plugins/compound-engineering/skills/ce-work/SKILL.md` — multi-phase skill with YAML frontmatter, `#$ARGUMENTS` input, parallel subagent dispatch - **Parallel dispatch**: `plugins/compound-engineering/skills/ce-review/SKILL.md` — spawns N reviewers in parallel, merges structured JSON results - **Subagent template**: `plugins/compound-engineering/skills/ce-review/references/subagent-template.md` — confidence rubric, false-positive suppression - **Codex delegation**: `plugins/compound-engineering/skills/ce-work-beta/SKILL.md` — `codex exec` stdin pipe, security posture, 3-failure auto-disable, environment guard - **Worktree management**: `plugins/compound-engineering/skills/git-worktree/SKILL.md` + `scripts/worktree-manager.sh` - **Scratch space**: `.context/compound-engineering//` with per-run subdirs for concurrent runs - **State file patterns**: YAML frontmatter in plan files, JSON schemas in ce:review references - **Skill-to-skill references**: `Load the skill` for pass-through; `/ce:compound` slash syntax for published commands ### Institutional Learnings - **State machine design is mandatory** for multi-phase workflows — re-read state after every transition, never carry stale values (`docs/solutions/skill-design/git-workflow-skills-need-explicit-state-machines-2026-03-27.md`) - **Script-first for measurement harnesses** — 60-75% token savings by moving mechanical work (parsing, classification, aggregation) into bundled scripts (`docs/solutions/skill-design/script-first-skill-architecture.md`) - **Confidence rubric pattern** — use 0.0-1.0 scale with explicit suppression threshold (0.60 proven in production), define false-positive categories (`ce:review subagent-template.md`) - **Pass paths not content to sub-agents** — orchestrator discovers paths, workers read what they need (`docs/solutions/skill-design/pass-paths-not-content-to-subagents-2026-03-26.md`) - **State transitions must be load-bearing** — if experiment states exist (proposed/running/measured/evaluated), at least one consumer must branch on them (`docs/solutions/workflow/todo-status-lifecycle.md`) - **Branch name sanitization** — `/` to `~` is injective for filesystem paths (`docs/solutions/developer-experience/branch-based-plugin-install-and-testing-2026-03-26.md`) ## Key Technical Decisions - **Linear keep/revert with parallel batches**: Each batch runs N experiments in parallel, best-in-batch is kept if it improves on current best, all others reverted. Simpler than tree search, compatible with git-native workflows. (see origin: Decision 1) - **Three-tier metrics**: Degenerate gates (fast, free, boolean) -> LLM-as-judge or hard primary metric -> diagnostics (logged only). Gates run first to avoid wasting judge calls on obviously broken solutions. (see origin: Decision 2) - **LLM-as-judge via stratified sampling**: ~30 samples per evaluation, stratified by output category (small/medium/large clusters), with user-defined rubric. Cost: ~$0.30-0.90 per experiment. Judge prompt is immutable (part of measurement harness). Judge score requires `minimum_improvement` (default 0.3 on a 1-5 scale) to accept as "better" — this accounts for sample-composition variance when output structure changes between experiments. (see origin: D4) - **Model-parsed spec, script-executed measurement**: The orchestrating agent reads and parses the YAML spec file directly (agents are natively capable of YAML handling). The measurement script receives flat arguments (command, timeout, working directory), runs the command, and returns raw JSON output. The agent evaluates gates and aggregates stability repeats. This follows the established plugin pattern where no shell scripts parse YAML — the model interprets structure, scripts handle I/O. - **Parallel-batch merge strategy**: When multiple experiments in a batch improve the metric: (1) Keep the best experiment, merge to optimization branch. (2) For each runner-up that also improved: check **file-level disjointness** with the kept experiment (same file modified by both = overlapping, even if different lines). (3) If disjoint: cherry-pick runner-up onto new baseline, re-run full measurement. (4) If combined measurement is strictly better: keep the cherry-pick. Otherwise revert and log as "promising alone but neutral/harmful in combination." (5) Process runners-up in descending metric order; stop after first failed combination. Config: `max_runner_up_merges_per_batch` (default: 1). Rationale: two changes that each independently improve a metric can interfere when combined (e.g., one tightens thresholds while another loosens them). This is expected, not a bug. - **Worktree isolation for parallel experiments**: Each experiment gets a git worktree under `.worktrees/` (aligned with existing convention) with copied shared resources. Codex sandboxes as opt-in alternative. Orchestrator retains git control. Max concurrent capped at 6 for worktree backend (git performance degrades beyond ~10-15 concurrent worktrees); 8+ only valid for Codex backend. (see origin: D6) - **Codex dispatch via stdin pipe**: Write prompt to temp file, pipe to `codex exec`, collect diff after completion. Security posture selected once per session. (see origin: D5) - **Context window management via rolling window + strategy digest**: The experiment log grows unboundedly (20-30 lines per experiment). The orchestrator does NOT read the full log each iteration. Instead: (1) maintain a rolling window of the last 10 experiments in working memory, (2) after each batch write a strategy digest summarizing what categories have been tried, what succeeded/failed, and the exploration frontier, (3) read the full log only in filtered sections (e.g., by category) when checking whether a specific hypothesis was already tried. The full log remains the durable ground truth on disk. - **Judge dispatch via batched parallel sub-agents**: Orchestrator selects samples per stratification config, groups them into batches of `judge.batch_size` (default: 10), dispatches `ceil(sample_size / batch_size)` parallel sub-agents. Each sub-agent evaluates its batch and returns structured JSON scores. Orchestrator aggregates. This follows the ce:review parallel reviewer dispatch pattern and avoids the overhead of spawning one sub-agent per sample. ## Open Questions ### Resolved During Planning - **Skill naming**: `ce-optimize` with directory `ce-optimize/`. The frontmatter name now matches the directory and slash command. - **Where does experiment state live**: `.context/compound-engineering/ce-optimize//` — contains spec, experiment log, strategy digest, and per-batch scratch. Cleaned after successful completion except the final experiment log which moves to the optimization branch. - **How are experiment branches named**: `optimize/` for the main optimization branch. Per-experiment worktree branches: `optimize//exp-`. Sanitized with `/` to `~` for filesystem paths. - **Judge model selection**: Haiku by default (fast, cheap), Sonnet optional. Specified in spec file. - **Who parses the YAML spec**: The orchestrating agent (model), not the measurement script. No CE scripts parse YAML — the established pattern is model reads structure, scripts handle I/O. The measurement script receives flat arguments and returns raw JSON. - **Judge dispatch mechanism**: Batched parallel sub-agents following ce:review pattern. Orchestrator selects samples, groups into batches of `judge.batch_size` (default: 10), dispatches parallel sub-agents, aggregates JSON scores. - **Branch collision on re-run**: Phase 0 detects existing `optimize/` branch and experiment log. Presents user with choice: resume (inherit existing state, continue from last iteration) or fresh start (archive old branch to `optimize//archived-`, clear log). - **Judge score comparability**: Add `judge.minimum_improvement` (default: 0.3 on 1-5 scale) as minimum improvement to accept. This accounts for sample-composition variance when output structure changes. Distinct from `noise_threshold` which handles run-to-run flakiness. ### Deferred to Implementation - **Exact gate check evaluation**: The agent interprets operator strings like `">= 0.85"` from the spec and evaluates them against metric values. The exact edge cases depend on what metric shapes users provide. - **Codex exec flag compatibility**: The exact `codex exec` flags may change. The skill should check `codex --version` and adapt. - **Worktree cleanup timing**: Whether to clean up worktrees immediately after each batch or defer to end-of-loop may depend on disk space constraints discovered at runtime. - **Harness bug discovered mid-loop**: If the measurement harness itself has a bug discovered during the loop, the user must fix it manually. The harness is immutable by design — the agent cannot modify it. After the fix, the user should re-baseline and resume (or start fresh). The exact UX for this depends on implementation. ## High-Level Technical Design > *This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.* ``` +-----------------+ | User provides | | goal + scope | +--------+--------+ | +--------v--------+ | Phase 0: Setup | | Create/load spec| +--------+--------+ | +--------v-----------+ | Phase 1: Scaffold | | Build/validate | | harness + baseline | | Probe parallelism | +--------+-----------+ | [USER GATE] | +--------v-----------+ | Phase 2: Hypotheses| | Generate + approve | | deps in bulk | +--------+-----------+ | +--------------v--------------+ | Phase 3: Optimize Loop | | | | +--- Batch N hypotheses | | | | | | +--+ Worktree/Codex | | | | | per experiment | | | | | implement | | | | | measure | | | | | collect metrics | | | +--+ | | | | | +--- Evaluate batch | | | gates -> judge -> rank | | | KEEP best / REVERT | | | | | +--- Update log + backlog | | +--- Check stop criteria | | +--- Next batch | +--------------+--------------+ | +--------v--------+ | Phase 4: Wrap-Up| | Summarize | | /ce:compound | | /ce:review | +--------+--------+ | [DONE] ``` ## Implementation Units ### Phase A: Reference Files and Scripts (no dependencies between units) - [ ] **Unit 1: Optimization spec schema** **Goal:** Define the YAML schema for the opt --- Source: [Claudary](https://claudary.paisolsolutions.com/skills/2026-03-29-001-feat-iterative-optimization-loop-skill-beta-plan) · https://claudary.paisolsolutions.com