Design Review Skill Benchmark

Date: 2026-04-07 Skill: kk:review-design (standard mode) Model: claude-opus-4-6 (1M context) Methodology: skill-creator eval framework — 3 test cases, each run with-skill and without-skill (baseline)

Summary

The skill achieves perfect format compliance across all 3 evals. Baselines detect substantive issues effectively but with freeform formatting. The skill's primary value is structural consistency and thoroughness, not raw issue detection — Claude already catches contradictions and gaps without guidance.

---

Test Cases

Eval 1: WIP Language-Specific Skills (Default Scope)

Prompt: /kk:review-design language-specific-skills Documents: docs/wip/language-specific-skills/design.md + implementation.md Purpose: Test the skill on a real, organic WIP feature with genuine codebase inconsistencies.

Scores

Assertions

With-Skill Output

## Design Review: Language-Specific Skills

**Scope:** design.md + implementation.md (default)
**Overall assessment:** CONCERNS_FOUND
**Documents:**

- Design: `/docs/wip/language-specific-skills/design.md` (224 lines)
- Implementation: `/docs/wip/language-specific-skills/implementation.md` (829 lines)

**Summary:** 9 findings: 2 critical, 3 high, 3 medium, 1 low

---

## Findings

### P0 - Critical

- **[INCONSISTENT]** Wrong directory path throughout both documents
  - **Section:** design.md:Architecture > Directory Structure; implementation.md:throughout
  - **Confidence:** 10/10 -- The design shows `.claude/skills/` as the skill directory
    (e.g., `.claude/skills/lang-go/`). The actual repository stores all skills under
    `klaude-plugin/skills/`. There is no `.claude/skills/` directory in the repository.
    Every path reference in both documents is wrong.
  - **Evidence:** design.md line 43 shows `.claude/skills/` tree; implementation.md lines
    26, 401-402, 423-424, 464, 470, 535, 564, 697, 701-702 all reference `.claude/skills/`.
    Actual skill location: `klaude-plugin/skills/`.
  - **Recommendation:** Replace all `.claude/skills/` references with the correct path. If
    the intent is that downstream template-created repos will have `.claude/skills/` (after
    plugin installation), this must be explicitly stated as an assumption in the design.

- **[INCONSISTENT]** LANGUAGE (singular) vs LANGUAGES (plural) mismatch with existing workflow
  - **Section:** design.md:Architecture > Template Cleanup Workflow; implementation.md:Phase 3
  - **Confidence:** 9/10 -- Both documents assume the workflow has a singular `LANGUAGE`
    input. The actual `template-cleanup.yml` workflow defines `LANGUAGES` (plural) as a
    comma-separated string input.
  - **Evidence:** Workflow file defines `LANGUAGES` (plural). Design doc line 111:
    "Accept LANGUAGE input". Implementation doc lines 393, 459, 489, 495: all reference
    `${{ inputs.LANGUAGE }}`.
  - **Recommendation:** Redesign the pruning logic to handle the existing `LANGUAGES`
    multi-value input.

### P1 - High

- **[INCONSISTENT]** References to non-existent `development-process` skill
  - **Section:** design.md:Architecture > Integration with Process Skills; implementation.md:Tasks 3.3, 4.2
  - **Confidence:** 9/10 -- The actual skill is named `development-guidelines`.
  - **Evidence:** implementation.md lines 424, 470, 564, 702 reference `development-process/SKILL.md`.
  - **Recommendation:** Replace all `development-process` references with `development-guidelines`.

- **[TECH_RISK]** Workflow modification approach is fragile -- inline YAML in implementation doc
  - **Section:** implementation.md:Task 3.2, 3.3, 3.4
  - **Confidence:** 8/10 -- Existing workflow delegates all logic to `template-cleanup.sh`.
  - **Evidence:** Workflow step: `run: chmod +x .github/scripts/template-cleanup.sh && ...`
  - **Recommendation:** Modify `template-cleanup.sh` instead of adding YAML steps.

- **[MISSING]** No consideration of the `klaude-plugin` distribution mechanism
  - **Section:** design.md (absent); implementation.md (absent)
  - **Confidence:** 8/10 -- Skills are distributed via the `klaude-plugin` package system.
  - **Evidence:** `settings.json` shows marketplace plugin configuration.
  - **Recommendation:** Add a section addressing distribution.

### P2 - Medium

- **[INCOMPLETE]** Template cleanup workflow logic does not handle multi-language projects
  - **Confidence:** 7/10
- **[INCOMPLETE]** Content depth guidelines are vague for topic files
  - **Confidence:** 6/10
- **[AMBIGUOUS]** Unclear relationship between `%LANGUAGE%` and `%LANGUAGE_LOWER%` placeholders
  - **Confidence:** 7/10

### P3 - Low

- **[STRUCTURE]** Typo in implementation doc ("well-establised")
  - **Confidence:** 10/10

---

## Clean Areas

- Design principles, language selection, topic file structure, non-goals,
  framework extensibility, phase structure, rollout strategy — all solid.

---

## Next Steps

I found 9 issues (P0: 2, P1: 3, P2: 3, P3: 1).

**How would you like to proceed?**

1. **Update docs** -- I'll revise the design docs to address all findings
2. **Update high severity only** -- Address P0/P1 issues
3. **Discuss specific items** -- Let's talk through particular findings
4. **Proceed to implementation** -- Findings are acceptable, move forward
5. **No changes** -- Review complete, no action needed

Baseline Output

# Design Review: Language-Specific Skills

**Documents reviewed:**
- `docs/wip/language-specific-skills/design.md`
- `docs/wip/language-specific-skills/implementation.md`

**Reviewer:** Claude (no skill guidance, natural review)

## Overall Assessment

The design is well-structured and clearly thought through. However, there are several
significant issues that need resolution before implementation, mostly stemming from
inconsistencies between the design docs and the actual codebase.

**Verdict:** Needs revision before implementation.

## Critical Issues

### 1. Wrong directory structure -- skills live in `klaude-plugin/skills/`, not `.claude/skills/`

The design repeatedly references `.claude/skills/` as the location for language skills.
In the actual repository, all skills live under `klaude-plugin/skills/`.

**Impact:** High. Must be corrected throughout both documents.

### 2. Workflow input mismatch: `LANGUAGE` (singular) vs `LANGUAGES` (plural)

The design assumes a `LANGUAGE` (singular) input parameter. The actual workflow uses
`LANGUAGES` (plural) and accepts comma-separated lists.

**Impact:** High. The entire template-cleanup integration needs redesign.

### 3. The `development-process` skill does not exist

The actual skill is called `development-guidelines`.

**Impact:** Medium. Easy to fix.

### 4. `%LANGUAGE%` placeholder handling is incomplete

**Impact:** Medium. Needs rethinking for multi-language.

## Moderate Issues

5. No consideration of the plugin distribution model
6. The cleanup script deletes everything outside specific directories
7. Implementation plan references a Phase 5 not mentioned in the phase list
8. Content depth estimates may be insufficient

## Minor Issues

9-12. Typo, inconsistent naming, optional migration guide, unmeasurable success metrics

## Recommendations

1. Fix all path references to use `klaude-plugin/skills/`
2. Redesign the multi-language story
3. Rename `development-process` to `development-guidelines`
4-7. Address distribution model, cleanup behavior, effort estimates, capy integration

Analysis

Both reviews caught the same critical issues (wrong paths, LANGUAGE/LANGUAGES mismatch, wrong skill name). The with-skill review adds:

• Structured severity classification with type codes
• Confidence scores with explicit reasoning for each finding
• Labeled Evidence and Recommendation fields
• Clean Areas section confirming what passed review
• Standardized Next Steps with 5 options

The baseline found 2 additional issues the skill missed (cleanup script deletion behavior, Phase 5 mismatch), suggesting the skill's structured workflow may cause slight tunnel vision. However, the baseline's free-form output lacks the consistency needed for programmatic consumption or cross-review comparison.

---

Eval 2: Synthetic Flawed Design (Issue Detection)

Prompt: /kk:review-design synthetic-flawed-feature Documents: docs/wip/synthetic-flawed-feature/design.md + implementation.md Purpose: Objective test with 6 planted flaws — measures detection accuracy rather than format compliance.

Planted Flaws

Scores

Assertions

With-Skill Output

## Design Review: synthetic-flawed-feature

**Scope:** design.md + implementation.md (default)
**Overall assessment:** MAJOR_GAPS

**Summary:** 11 findings: 2 critical, 4 high, 3 medium, 2 low

### P0 - Critical

- **[INCONSISTENT]** Endpoint count mismatch between design and implementation
  - **Confidence:** 10/10 -- Design says "3 REST API endpoints" in overview, defines 2.
    Implementation defines 4. All three numbers disagree.
  - **Evidence:** design.md overview: "3 REST API endpoints"; body defines POST /send and
    GET /preferences; implementation Step 5 lists 4 endpoints.

- **[INCONSISTENT]** Push notification technology contradicts between documents
  - **Confidence:** 10/10 -- design.md specifies FCM. implementation.md says
    "WebSocket for push (replacing FCM as noted in the design)" -- but the design says
    no such thing.

### P1 - High

- **[INCONSISTENT]** Database technology contradicts (PostgreSQL vs MongoDB)
  - **Confidence:** 10/10

- **[INCOMPLETE]** Missing endpoint definitions in design.md
  - **Confidence:** 9/10

- **[MISSING]** No error handling or failure response specification
  - **Confidence:** 8/10

- **[TECH_RISK]** WebSocket push has unaddressed offline delivery gap
  - **Confidence:** 8/10

### P2 - Medium

- **[AMBIGUOUS]** Filtering section references undefined endpoint
- **[INCOMPLETE]** Rate limiting lacks specification
- **[INCOMPLETE]** Testing strategy is a single sentence

### P3 - Low

- **[STRUCTURE]** Implementation overview contradicts its own content
- **[MISSING]** No monitoring or observability mentioned

## Clean Areas

- Security model, async delivery architecture, component separation, file structure,
  external service selection -- all sound.

Baseline Output

# Design Review: Notification Service

The documents contain multiple significant inconsistencies, gaps, and underspecified
areas that would block a clean implementation.

## Critical Issues

1. Push notification technology contradiction (FCM vs WebSocket)
2. Database technology contradiction (PostgreSQL vs MongoDB)
3. Missing API endpoint in design (2 of 3 documented)
4. Endpoint count mismatch (3 vs 2 vs 4)

## Major Issues

5. WebSocket infrastructure appears only in implementation
6. Notification history endpoint is underspecified
7. Dead letter queue mentioned but not designed
8. Rate limiting is vague

## Minor Issues

9-14. No response schemas, no error handling, opaque payload, minimal testing,
no deployment details, thin security

## Document Quality Assessment

| Criterion | Rating |
|-----------|--------|
| Completeness | Poor |
| Internal consistency | Poor |
| Cross-doc consistency | Poor |
| Technical soundness | Fair |
| Implementability | Poor |

Analysis

Both reviews detected all 6 planted flaws. The baseline found 14 issues vs the skill's 11, including extras the skill missed (dead letter queue design gap, no response schemas, opaque payload field, no deployment details, thin security). The baseline is slightly more thorough on content.

The +22% delta is the smallest across all evals because this eval's assertions are weighted toward content detection (6 of 9 assertions). The 2 format-only assertions (P0/P1 labels, structured fields) account for the entire gap.

Key insight: For issue detection, the skill provides no meaningful advantage over baseline Claude. Its value is entirely in standardized output structure.

---

Eval 3: Completed Capy-Integration (All Scope)

Prompt: `Review the design docs for the capy-integration feature. The docs are at