gstack development

Commands

bun install          # install dependencies
bun test             # run free tests (browse + snapshot + skill validation)
bun run test:evals   # run paid evals: LLM judge + E2E (diff-based, ~$4/run max)
bun run test:evals:all  # run ALL paid evals regardless of diff
bun run test:gate    # run gate-tier tests only (CI default, blocks merge)
bun run test:periodic  # run periodic-tier tests only (weekly cron / manual)
bun run test:e2e     # run E2E tests only (diff-based, ~$3.85/run max)
bun run test:e2e:all # run ALL E2E tests regardless of diff
bun run eval:select  # show which tests would run based on current diff
bun run dev <cmd>    # run CLI in dev mode, e.g. bun run dev goto https://example.com
bun run build        # gen docs + compile binaries
bun run gen:skill-docs  # regenerate SKILL.md files from templates
bun run skill:check  # health dashboard for all skills
bun run dev:skill    # watch mode: auto-regen + validate on change
bun run eval:list    # list all eval runs from ~/.gstack-dev/evals/
bun run eval:compare # compare two eval runs (auto-picks most recent)
bun run eval:summary # aggregate stats across all eval runs
bun run slop          # full slop-scan report (all files)
bun run slop:diff     # slop findings in files changed on this branch only

test:evals requires ANTHROPIC_API_KEY. Codex E2E tests (test/codex-e2e.test.ts) use Codex's own auth from ~/.codex/ config — no OPENAI_API_KEY env var needed.

Where the keys live on this machine. Conductor workspaces don't inherit the user's interactive shell env, so ANTHROPIC_API_KEY and OPENAI_API_KEY aren't in the default process env. Before running any paid eval / E2E, source them from ~/.zshrc (that's where Garry keeps them):

bash -c '
  eval "$(grep -E "^export (ANTHROPIC_API_KEY|OPENAI_API_KEY)=" ~/.zshrc)"

  EVALS=1 EVALS_TIER=periodic bun test test/skill-e2e-<whatever>.test.ts
'

Do not echo the key value anywhere (stdout, logs, shell history). The grep+eval pattern keeps it in process env only. When passing to a test's Agent SDK, do NOT pass env: {...} to runAgentSdkTest — the SDK's auth pipeline doesn't pick up the key the same way when env is supplied as an object (confirmed failure mode). Instead, mutate process.env.ANTHROPIC_API_KEY ambiently before the call and restore in finally. E2E tests stream progress in real-time (tool-by-tool via --output-format stream-json --verbose). Results are persisted to ~/.gstack-dev/evals/ with auto-comparison against the previous run.

Diff-based test selection: test:evals and test:e2e auto-select tests based on git diff against the base branch. Each test declares its file dependencies in test/helpers/touchfiles.ts. Changes to global touchfiles (session-runner, eval-store, touchfiles.ts itself) trigger all tests. Use EVALS_ALL=1 or the :all script variants to force all tests. Run eval:select to preview which tests would run.

Two-tier system: Tests are classified as gate or periodic in E2E_TIERS (in test/helpers/touchfiles.ts). CI runs only gate tests (EVALS_TIER=gate); periodic tests run weekly via cron or manually. Use EVALS_TIER=gate or EVALS_TIER=periodic to filter. When adding new E2E tests, classify them:

1. Safety guardrail or deterministic functional test? -> gate
2. Quality benchmark, Opus model test, or non-deterministic? -> periodic
3. Requires external service (Codex, Gemini)? -> periodic

Testing

bun test             # run before every commit — free, <2s
bun run test:evals   # run before shipping — paid, diff-based (~$4/run max)

bun test runs skill validation, gen-skill-docs quality checks, and browse integration tests. bun run test:evals runs LLM-judge quality evals and E2E tests via claude -p. Both must pass before creating a PR.

Project structure

gstack/
├── browse/          # Headless browser CLI (Playwright)
│   ├── src/         # CLI + server + commands
│   │   ├── commands.ts  # Command registry (single source of truth)
│   │   └── snapshot.ts  # SNAPSHOT_FLAGS metadata array
│   ├── test/        # Integration tests + fixtures
│   └── dist/        # Compiled binary
├── hosts/           # Typed host configs (one per AI agent)
│   ├── claude.ts    # Primary host config
│   ├── codex.ts, factory.ts, kiro.ts  # Existing hosts
│   ├── opencode.ts, slate.ts, cursor.ts, openclaw.ts  # IDE hosts
│   ├── hermes.ts, gbrain.ts  # Agent runtime hosts
│   └── index.ts     # Registry: exports all, derives Host type
├── scripts/         # Build + DX tooling
│   ├── gen-skill-docs.ts  # Template → SKILL.md generator (config-driven)
│   ├── host-config.ts     # HostConfig interface + validator
│   ├── host-config-export.ts  # Shell bridge for setup script
│   ├── host-adapters/     # Host-specific adapters (OpenClaw tool mapping)
│   ├── resolvers/   # Template resolver modules (preamble, design, review, gbrain, etc.)
│   ├── skill-check.ts     # Health dashboard
│   └── dev-skill.ts       # Watch mode
├── test/            # Skill validation + eval tests
│   ├── helpers/     # skill-parser.ts, session-runner.ts, llm-judge.ts, eval-store.ts
│   ├── fixtures/    # Ground truth JSON, planted-bug fixtures, eval baselines
│   ├── skill-validation.test.ts  # Tier 1: static validation (free, <1s)
│   ├── gen-skill-docs.test.ts    # Tier 1: generator quality (free, <1s)
│   ├── skill-llm-eval.test.ts   # Tier 3: LLM-as-judge (~$0.15/run)
│   └── skill-e2e-*.test.ts       # Tier 2: E2E via claude -p (~$3.85/run, split by category)
├── qa-only/         # /qa-only skill (report-only QA, no fixes)
├── plan-design-review/  # /plan-design-review skill (report-only design audit)
├── design-review/    # /design-review skill (design audit + fix loop)
├── ship/            # Ship workflow skill
├── review/          # PR review skill
├── plan-ceo-review/ # /plan-ceo-review skill
├── plan-eng-review/ # /plan-eng-review skill
├── autoplan/        # /autoplan skill (auto-review pipeline: CEO → design → eng)
├── benchmark/       # /benchmark skill (performance regression detection)
├── canary/          # /canary skill (post-deploy monitoring loop)
├── codex/           # /codex skill (multi-AI second opinion via OpenAI Codex CLI)
├── land-and-deploy/ # /land-and-deploy skill (merge → deploy → canary verify)
├── office-hours/    # /office-hours skill (YC Office Hours — startup diagnostic + builder brainstorm)
├── investigate/     # /investigate skill (systematic root-cause debugging)
├── retro/           # Retrospective skill (includes /retro global cross-project mode)
├── bin/             # CLI utilities (gstack-repo-mode, gstack-slug, gstack-config, etc.)
├── document-release/ # /document-release skill (post-ship doc updates)
├── cso/             # /cso skill (OWASP Top 10 + STRIDE security audit)
├── design-consultation/ # /design-consultation skill (design system from scratch)
├── design-shotgun/  # /design-shotgun skill (visual design exploration)
├── open-gstack-browser/  # /open-gstack-browser skill (launch GStack Browser)
├── connect-chrome/  # symlink → open-gstack-browser (backwards compat)
├── design/          # Design binary CLI (GPT Image API)
│   ├── src/         # CLI + commands (generate, variants, compare, serve, etc.)
│   ├── test/        # Integration tests
│   └── dist/        # Compiled binary
├── extension/       # Chrome extension (side panel + activity feed + CSS inspector)
├── lib/             # Shared libraries (worktree.ts)
├── docs/designs/    # Design documents
├── setup-deploy/    # /setup-deploy skill (one-time deploy config)
├── .github/         # CI workflows + Docker image
│   ├── workflows/   # evals.yml (E2E on Ubicloud), skill-docs.yml, actionlint.yml
│   └── docker/      # Dockerfile.ci (pre-baked toolchain + Playwright/Chromium)
├── contrib/         # Contributor-only tools (never installed for users)
│   └── add-host/    # /gstack-contrib-add-host skill
├── setup            # One-time setup: build binary + symlink skills
├── SKILL.md         # Generated from SKILL.md.tmpl (don't edit directly)
├── SKILL.md.tmpl    # Template: edit this, run gen:skill-docs
├── ETHOS.md         # Builder philosophy (Boil the Lake, Search Before Building)
└── package.json     # Build scripts for browse

SKILL.md workflow

SKILL.md files are generated from .tmpl templates. To update docs:

1. Edit the .tmpl file (e.g. SKILL.md.tmpl or browse/SKILL.md.tmpl)
2. Run bun run gen:skill-docs (or bun run build which does it automatically)
3. Commit both the .tmpl and generated .md files

To add a new browse command: add it to browse/src/commands.ts and rebuild. To add a snapshot flag: add it to SNAPSHOT_FLAGS in browse/src/snapshot.ts and rebuild.

Token ceiling: Generated SKILL.md files trip a warning above 160KB (~40K tokens). This is a "watch for feature bloat" guardrail, not a hard gate. Modern flagship models have 200K-1M context windows, so 40K is 4-20% of window, and prompt caching makes the marginal cost of larger skills small. The ceiling exists to catch runaway preamble/resolver growth, not to force compression on carefully-tuned big skills (ship, plan-ceo-review, office-hours legitimately pack 25-35K tokens of behavior). If you blow past 40K, the right fix is usually: (1) look at WHAT grew, (2) if one resolver added 10K+ in a single PR, question whether it belongs inline or as a reference doc, (3) only compress carefully-tuned prose as a last resort — cuts to the coverage audit, review army, or voice directive have real quality cost.

Merge conflicts on SKILL.md files: NEVER resolve conflicts on generated SKILL.md files by accepting either side. Instead: (1) resolve conflicts on the .tmpl templates and scripts/gen-skill-docs.ts (the sources of truth), (2) run bun run gen:skill-docs to regenerate all SKILL.md files, (3) stage the regenerated files. Accepting one side's generated output silently drops the other side's template changes.

Platform-agnostic design

Skills must NEVER hardcode framework-specific commands, file patterns, or directory structures. Instead:

1. Read CLAUDE.md for project-specific config (test commands, eval commands, etc.)
2. If missing, AskUserQuestion — let the user tell you or let gstack search the repo
3. Persist the answer to CLAUDE.md so we never have to ask again

This applies to test commands, eval commands, deploy commands, and any other project-specific behavior. The project owns its config; gstack reads it.

Writing SKILL templates

SKILL.md.tmpl files are prompt templates read by Claude, not bash scripts. Each bash code block runs in a separate shell — variables do not persist between blocks.

Rules:

• Use natural language for logic and state. Don't use shell variables to pass state between code blocks. Instead, tell Claude what to remember and reference it in prose (e.g., "the base branch detected in Step 0").
• Don't hardcode branch names. Detect main/master/etc dynamically via gh pr view or gh repo view. Use {{BASE_BRANCH_DETECT}} for PR-targeting skills. Use "the base branch" in prose, <base> in code block placeholders.
• Keep bash blocks self-contained. Each code block should work independently. If a block needs context from a previous step, restate it in the prose above.
• Express conditionals as English. Instead of nested if/elif/else in bash, write numbered decision steps: "1. If X, do Y. 2. Otherwise, do Z."

Writing style (V1)

Default output from every tier-≥2 skill follows the Writing Style section in scripts/resolvers/preamble.ts: jargon glossed on first use (curated list in scripts/jargon-list.json, baked at gen-skill-docs time), questions framed in outcome terms ("what breaks for your users if...") not implementation terms, short sentences, decisions close with user impact. Power users who want the tighter V0 prose set gstack-config set explain_level terse (binary switch, no middle mode). See docs/designs/PLAN_TUNING_V1.md for the full design rationale. The review pacing overhaul that originally tried to ride alongside writing-style was extracted to V1.1 — see docs/designs/PACING_UPDATES_V0.md.

Browser interaction

When you need to interact with a browser (QA, dogfooding, cookie setup), use the /browse skill or run the browse binary directly via $B <command>. NEVER use mcp__claude-in-chrome__* tools — they are slow, unreliable, and not what this project uses.

Sidebar architecture: Before modifying sidepanel.js, background.js, content.js, sidebar-agent.ts, or sidebar-related server endpoints, read docs/designs/SIDEBAR_MESSAGE_FLOW.md. It documents the full initialization timeline, message flow, auth token chain, tab concurrency model, and known failure modes. The sidebar spans 5 files across 2 codebases (extension + server) with non-obvious ordering dependencies. The doc exists to prevent the kind of silent failures that come from not understanding the cross-component flow.

Transport-layer security (v1.6.0.0+). When pair-agent starts an ngrok tunnel, the daemon binds two HTTP listeners: a local listener (127.0.0.1, full command surface, never forwarded) and a tunnel listener (locked allowlist: /connect, /command with a scoped token + 17-command browser-driving allowlist, /sidebar-chat). ngrok forwards only the tunnel port. Root tokens over the tunnel return 403. SSE endpoints use a 30-minute HttpOnly gstack_sse cookie minted via POST /sse-session (never valid against /command). Tunnel-surface rejections go to ~/.gstack/security/attempts.jsonl via tunnel-denial-log.ts. Before editing server.ts, sse-session-cookie.ts, or tunnel-denial-log.ts, read ARCHITECTURE.md — the module boundary (no imports from token-registry.ts into sse-session-cookie.ts) is load-bearing for scope isolation.

Sidebar security stack (layered defense against prompt injection):

Critical constraint: security-classifier.ts CANNOT be imported from the compiled browse binary. @huggingface/transformers v4 requires onnxruntime-node which fails to dlopen from Bun compile's temp extract dir. Only security.ts (pure-string operations — canary, verdict combiner, attack log, status) is safe for server.ts. See `~/.gstack/projects/garrytan-gstack/ceo