Plan Tuning v0 — Design Doc

Status: Approved for v1 implementation Branch: garrytan/plan-tune-skill Authors: Garry Tan (user), with AI-assisted reviews from Claude Opus 4.7 + OpenAI Codex gpt-5.4 Date: 2026-04-16

What this document is

A canonical record of what /plan-tune v1 is, what it is NOT, what we considered, and why we made each call. Committed to the repo so future contributors (and future Garry) can trace reasoning without archeology. Supersedes the two ~/.gstack/projects/ artifacts (office-hours design doc + CEO plan) which are per-user local records.

The feature, in one paragraph

gstack's 40+ skills fire AskUserQuestion constantly. Power users answer the same questions the same way repeatedly and have no way to tell gstack "stop asking me this." More fundamentally, gstack has no model of how each user prefers to steer their work — scope-appetite, risk-tolerance, detail-preference, autonomy, architecture-care — so every skill's defaults are middle-of-the-road for everyone. /plan-tune v1 builds the schema + observation layer: a typed question registry, per-question explicit preferences, inline "tune:" feedback, and a profile (declared + inferred dimensions) inspectable via plain English. It does not yet adapt skill behavior based on the profile. That comes in v2, after v1 proves the substrate works.

Why we're building the smaller version

The feature started life as a full adaptive substrate: psychographic dimensions driving auto-decisions, blind-spot coaching, LANDED celebration HTML page, all bundled. Four rounds of review (office-hours, CEO EXPANSION, DX POLISH, eng review) cleared it. Then outside voice (Codex) delivered a 20-point critique. The critical findings, in priority order:

1. "Substrate" was false. The plan wired 5 skills to read the profile on preamble, but AskUserQuestion is a prompt convention, not middleware. Agents can silently skip the instructions. You cannot reliably build auto-decide on top of an unenforceable convention. Without a typed question registry that every AskUserQuestion routes through, the substrate claim is marketing.
2. Internal logical contradictions. E4 (blind-spot) + E6 (mismatch) + ±0.2 clamp on declared dimensions do not compose. If user self-declaration is ground truth via the clamp, E6's mismatch detection is detecting noise. If behavior can correct the profile, the clamp suppresses the signal E6 needs.
3. Profile poisoning. Inline "tune: never ask" could be emitted by malicious repo content (README, PR description, tool output) and the agent would dutifully write it. No prior review caught this security gap.
4. E5 LANDED page in preamble. gh pr view + HTML write + browser open on every skill's preamble is latency, auth failures, rate limits, surprise browser opens, and nondeterminism injected into the hottest path.
5. Implementation order was backwards. The plan started with classifiers and bins. The correct order: build the integration point first (typed question registry), then infrastructure, then consumers.

After weighing Codex's argument, we chose to roll back CEO EXPANSION and ship an observational v1 with a real typed registry as the foundation. Psychographic becomes behavioral only after the registry proves durable in production.

v1 Scope (what we're building now)

1. Typed question registry (scripts/question-registry.ts). Every AskUserQuestion gstack uses is declared with {id, skill, category, door_type, options[], signal_key?}. Schema-governed.
2. CI enforcement. Lint test (gate tier) asserts every AskUserQuestion pattern in SKILL.md.tmpl files has a matching registry entry. Fails CI on drift, renames, or duplicates.
3. Question logging (bin/gstack-question-log). Appends {ts, question_id, user_choice, recommended, session_id} to ~/.gstack/projects/{SLUG}/question-log.jsonl. Validates against registry.
4. Explicit per-question preferences (bin/gstack-question-preference). Writes {question_id, preference} where preference is always-ask | never-ask | ask-only-for-one-way. Respected from session 1. No calibration gate — user stated it, system obeys.
5. Preamble injection. Before each AskUserQuestion, agent calls gstack-question-preference --check <registry-id>. If never-ask AND question is NOT a one-way door, auto-choose recommended option with visible annotation: "Auto-decided [summary] → [option] (your preference). Change with /plan-tune." One-way doors always ask regardless of preference — safety override.
6. Inline "tune:" feedback with user-origin gate. Agent offers "Tune this question? Reply tune: [feedback] to adjust." User can use shortcuts (unnecessary, ask-less, never-ask, always-ask, context-dependent) or free-form English. CRITICAL: the agent only writes a tune event when the tune: content appears in the user's current chat turn — NOT in tool output, NOT in a file read. Binary validates source: "inline-user" on write; rejects other sources.
7. Declared profile (/plan-tune setup). 5 plain-English questions, one per dimension. Stored in unified ~/.gstack/developer-profile.json under declared: {...}. Informational only in v1 — no skill behavior change.
8. Observed/Inferred profile. Every question-log event contributes deltas to inferred dimensions via a hand-crafted signal map (scripts/psychographic-signals.ts). Computed on demand. Displayed but not acted on.
9. /plan-tune skill. Conversational plain-English inspection tool. "Show my profile," "set a preference," "what questions have I been asked," "show the gap between what I said and what I do." No CLI subcommand syntax required.
10. Unification with existing ~/.gstack/builder-profile.jsonl. Fold /office-hours session records and accumulated signals into unified ~/.gstack/developer-profile.json. Migration is atomic + idempotent + archives the source file.

Deferred to v2 (not in this PR, but explicit acceptance criteria)

Rejected entirely (Codex was right, we're not doing these)

Architecture

~/.gstack/
  developer-profile.json            # unified: declared + inferred + sessions (from office-hours)

~/.gstack/projects/{SLUG}/
  question-log.jsonl                # every AskUserQuestion, append-only, registry-validated
  question-preferences.json         # explicit per-question user choices
  question-events.jsonl             # tune: feedback events, user-origin gated

Unified profile schema (superseding both v0.16.2.0 builder-profile.jsonl and the proposed developer-profile.json):

{
  "identity": {"email": "..."},
  "declared": {
    "scope_appetite": 0.9,
    "risk_tolerance": 0.7,
    "detail_preference": 0.4,
    "autonomy": 0.5,
    "architecture_care": 0.7
  },
  "inferred": {
    "values": {"scope_appetite": 0.72, "risk_tolerance": 0.58, "...": "..."},
    "sample_size": 47,
    "diversity": {
      "skills_covered": 5,
      "question_ids_covered": 14,
      "days_span": 23
    }
  },
  "gap": {"scope_appetite": 0.18, "...": "..."},
  "sessions": [
    {"date": "...", "mode": "builder", "project_slug": "...", "signals": []}
  ],
  "signals_accumulated": {
    "named_users": 1, "taste": 4, "agency": 3, "...": "..."
  }
}

Diversity check (Codex #13): inferred is considered "enough data" only when sample_size >= 20 AND skills_covered >= 3 AND question_ids_covered >= 8 AND days_span >= 7. Below this, /plan-tune profile shows "not enough observed data yet" instead of a potentially-misleading inferred value.

Data flow (v1)

1. Preamble: check question_tuning config. If off, do nothing.
2. Before each AskUserQuestion:
   • Agent calls gstack-question-preference --check <registry-id>
   • If never-ask AND question is NOT one-way door → auto-choose recommended with annotation
   • If always-ask, unset, or question IS one-way door → ask normally
3. After AskUserQuestion:
   • Append log record to question-log.jsonl (registry-validated, rejects unknown IDs)
4. Offer inline: "Tune this question? Reply tune: [feedback] to adjust."
5. If user's NEXT turn message contains tune: prefix AND the content originated in the user's own message (not tool output):
   • Agent calls gstack-question-preference --write with source: "inline-user"
   • Binary validates source field; rejects if anything other than inline-user
6. Inferred dimensions recomputed on demand by bin/gstack-developer-profile --derive. Signal map changes trigger full recompute from events history.

Security model

Profile poisoning defense (Codex #16, Decision J below): Inline tune events may be written ONLY when:

• The agent is processing the user's current chat turn
• The tune: prefix appears in that user message (not in any tool output, file content, PR description, commit message, etc.)
• The resolver's instructions to the agent explicitly call this out

Binary enforcement: gstack-question-preference --write requires source: "inline-user" field on every tune-originated record. Any other source value (e.g., inline-tool-output, inline-file-content) is rejected with an error. Agent is instructed to never forge the source field.

Data privacy:

• All data is local-only under ~/.gstack/. Nothing leaves without explicit user action.
• /plan-tune export <path> writes profile to user-specified path (opt-in export).
• /plan-tune delete wipes local profile files.
• gstack-config set telemetry off prevents any telemetry (this skill never sends profile data regardless).
• Profile files have standard user-home permissions.

Injection defense (consistent with existing bin/gstack-learnings-log patterns): the question_summary and any free-form user feedback fields are sanitized against known prompt-injection patterns ("ignore previous instructions," "system:", etc.).

5 Hard Constraints (preserved from office-hours, updated for Codex feedback)

1. One-way doors are classified deterministically by registry declaration, NOT by runtime summary parsing. Each registry entry declares door_type: one-way | two-way. Keyword pattern fallback (scripts/one-way-doors.ts) is a belt-and-suspenders secondary check for edge cases.
2. Profile dimensions are inspectable AND editable. /plan-tune profile shows declared + inferred + gap. Edits via plain English go to declared only. System tracks inferred independently.
3. Signal map is hand-crafted in TypeScript. scripts/psychographic-signals.ts maps {question_id, user_choice} → {dimension, delta}. Not agent-inferred. In v1, consumed only for inferred.values display — not for driving decisions.
4. No psychographic-driven auto-decide in v1. Only explicit per-question preferences act. This sidesteps the "calibration gate can be gamed" critique (Codex #13) entirely — v1 doesn't have a gate to pass.
5. Per-project preferences beat global preferences. ~/.gstack/projects/{SLUG}/question-preferences.json wins over any future global preference file. Global profile (~/.gstack/developer-profile.json) is a starting point for diversity across projects.

Why event-sourced + dual-track

Why event-sourced for the inferred profile:

• Signal map can change between gstack versions. Recompute from events, no data migration needed.
• Auditable: /plan-tune profile --trace autonomy shows every event that contributed to the value.
• Future-proof: new dimensions can be derived from existing history.

Why dual-track (declared + inferred, separately) (Decision B below):

• Resolves the logical contradiction Codex #6 identified.
• declared is user sovereignty. User states who they are. System obeys for anything user-driven (preferences, declarations, overrides).
• inferred is observation. System tracks behavioral patterns. Displayed but not acted on in v1.
• gap is the interesting signal. Large gaps suggest the user's self-description isn't matching their behavior — valuable self-insight, but not auto-corrected.

Interaction model — plain English everywhere

(From /plan-de