Discovery Workflow

Methodology for analyzing the Testing Handbook and identifying skill candidates.

Quick Navigation:

• Phase 0: Locate Handbook
• Phase 1: Handbook Analysis
• Phase 2: Plan Generation
• Phase 3: Prepare Generation Context

Progress Tracking

Use TodoWrite throughout discovery to track progress and give visibility to the user:

Discovery phase todos:
- [ ] Locate handbook repository
- [ ] Scan /fuzzing/ sections
- [ ] Scan /static-analysis/ sections
- [ ] Scan /crypto/ sections
- [ ] Scan /web/ sections
- [ ] Build candidate list with types
- [ ] Resolve conflicts and edge cases
- [ ] Present plan to user
- [ ] Await user approval

Mark each todo as in_progress when starting and completed when done. This helps users understand where you are in the process.

Phase 0: Locate Handbook

Before analysis, locate or obtain the Testing Handbook repository.

Step 1: Check Common Locations

# Check common locations (simple version)
for dir in ./testing-handbook ../testing-handbook ~/testing-handbook; do
  [ -d "$dir/content/docs" ] && handbook_path="$dir" && echo "✓ Found: $dir" && break
done
[ -z "$handbook_path" ] && echo "Handbook not found in common locations"

Step 2: Ask User (if not found)

If handbook not found in common locations:

"Where is the Testing Handbook repository located? (full path)"

Step 3: Clone as Last Resort (if user agrees)

git clone --depth=1 https://github.com/trailofbits/testing-handbook.git
handbook_path="./testing-handbook"

Set handbook_path Variable

Once located, set handbook_path and use it for all subsequent paths:

handbook_path="/path/to/testing-handbook"  # Set to actual location

All paths below are relative to {handbook_path}.

Error Recovery

Phase 1: Handbook Analysis

1.1 Scan Directory Structure

Scan the handbook at:

{handbook_path}/content/docs/

Directory structure pattern:

docs/
├── fuzzing/
│   ├── _index.md              # Section overview
│   ├── c-cpp/
│   │   ├── _index.md          # Language subsection
│   │   ├── 10-libfuzzer/      # Tool directory
│   │   │   └── index.md
│   │   ├── 11-aflpp/
│   │   └── techniques/        # Shared techniques
│   └── rust/
├── static-analysis/
│   ├── semgrep/
│   │   ├── _index.md
│   │   ├── 00-installation.md
│   │   ├── 10-advanced.md
│   │   ├── 20-ci.md
│   │   └── 99-resources.md
│   └── codeql/
├── crypto/
│   ├── wycheproof/
│   └── constant_time_tool/
└── web/
    └── burp/

1.2 Parse Frontmatter

Each markdown file has YAML frontmatter:

---
title: "Semgrep"
weight: 2
summary: "Fast static analysis for finding bugs..."
bookCollapseSection: true
draft: false                    # Check this!
---

Key fields:

1.3 Identify Skill Candidates

Decision Table:

For each directory found, apply the first matching rule:

Hard Exclusions (GUI-only tools):

Some handbook sections describe tools that require graphical user interfaces and cannot be operated by Claude. Skip these unconditionally:

These tools are excluded because Claude cannot:

• Launch or interact with GUI applications
• Click buttons, navigate menus, or view visual elements
• Operate browser-based or desktop UI tools

For web security testing, prefer CLI-based alternatives documented elsewhere (e.g., curl, httpie, custom scripts).

Classification priority (when multiple patterns match):

1. Most specific path wins (deeper = higher priority)
2. Type preference: Tool > Fuzzer > Technique > Domain
3. When in doubt, flag for user review in the plan

Examples:

1.4 Build Candidate List

For each candidate, extract:

- name: libfuzzer
  type: fuzzer
  source: /docs/fuzzing/c-cpp/10-libfuzzer/
  summary: "libFuzzer is a coverage-guided fuzzer..."
  weight: 1
  has_resources: true           # Has 99-resources.md
  related_sections:
    - /docs/fuzzing/techniques/01-writing-harnesses/
    - /docs/fuzzing/techniques/03-asan/

1.5 Candidate Prioritization

When many candidates exist, prioritize by:

Conflict resolution:

• If multiple patterns match a section, use the most specific (deepest path wins)
• If a section could be multiple types, prefer: Tool > Fuzzer > Technique > Domain
• When in doubt, flag for user review in the plan

Phase 2: Plan Generation

2.1 Plan Format

Present to user in this format:

# Skill Generation Plan

## Summary
- **Handbook analyzed:** {handbook_path}
- **Total sections:** 25
- **Skills to generate:** 8
- **Sections to skip:** 2

---

## Skills to Generate

| # | Skill Name | Source Section | Type | Related Sections |
|---|------------|----------------|------|------------------|
| 1 | libfuzzer | /fuzzing/c-cpp/10-libfuzzer/ | Fuzzer | techniques/asan, techniques/harness |
| 2 | aflpp | /fuzzing/c-cpp/11-aflpp/ | Fuzzer | techniques/asan |
| 3 | cargo-fuzz | /fuzzing/rust/10-cargo-fuzz/ | Fuzzer | - |
| 4 | wycheproof | /crypto/wycheproof/ | Domain | - |
| 5 | fuzz-harness-writing | /fuzzing/techniques/01-writing-harnesses/ | Technique | - |
| 6 | coverage-analysis | /fuzzing/c-cpp/techniques/01-coverage/ | Technique | - |
| 7 | address-sanitizer | /fuzzing/techniques/03-asan/ | Technique | - |

---

## Skipped Sections

| Section | Reason |
|---------|--------|
| /docs/dynamic-analysis/ | `draft: true` in frontmatter |
| /docs/fuzzing/3-python.md | Single file, insufficient content |
| /docs/web/burp/ | GUI-only tool (excluded) |

---

## External Resources to Fetch

| Section | Resource Count | Source File |
|---------|---------------|-------------|
| Fuzzing | 5 | /fuzzing/91-resources.md |
| Semgrep | 3 | /static-analysis/semgrep/99-resources.md |
| CodeQL | 4 | /static-analysis/codeql/99-resources.md |

---

## Actions

- [ ] Confirm plan and proceed with generation
- [ ] Modify: Remove skill #X from plan
- [ ] Modify: Change skill #Y type
- [ ] Cancel generation

Make the actions navigable and selectable if possible using built-in tool like TodoWrite.

2.2 User Interaction

After presenting plan:

1. Wait for user confirmation or modifications
2. Apply any modifications to plan
3. Proceed with generation only after explicit approval

Acceptable modifications:

• Remove skills from plan
• Change skill type
• Skip updates
• Add custom related sections
• Change skill names

Phase 3: Prepare Generation Context

This phase prepares everything needed for generation agents. It combines content aggregation with agent handoff preparation.

3.1 Collect Content Per Skill

For each approved skill, collect content from:

1. Primary section: Main _index.md or index.md
2. Numbered files: 00-installation.md, 10-advanced.md, etc.
3. Related sections: As specified in related_sections
4. Resources: From 99-resources.md (titles only)

3.2 Content Processing Rules

Content processing rules are defined in the agent prompt template.

Authoritative source: agent-prompt.md

The agent prompt contains:

• Hugo shortcode conversion table
• Code block preservation rules
• Image/video handling
• YAML frontmatter requirements
• Pre-write validation checklist
• Line count splitting rules (when to split large skills)

Do not duplicate these rules here. Generation agents receive them via the prompt template.

3.3 External Resources

Extract from 99-resources.md or 91-resources.md:

## Resources

- [Introduction to Semgrep](https://www.youtube.com/watch?v=...) - Trail of Bits Webinar
- [Semgrep Documentation](https://semgrep.dev/docs/) - Official docs
- [Custom Rules Guide](https://semgrep.dev/docs/writing-rules/) - Rule authoring

For non-video resources (documentation, blogs, guides):

• Use WebFetch to retrieve content
• Extract key insights, techniques, code examples
• Summarize actionable information for the skill
• Include attribution with URL

For video resources (YouTube, Vimeo, etc.):

• Extract title and URL only
• Do NOT attempt to fetch video content
• Include brief description if available in handbook

Video URL patterns to skip fetching:

• youtube.com, youtu.be
• vimeo.com
• *.mp4, *.webm, *.mov direct links
• twitch.tv, dailymotion.com

3.4 Edge Case Handling

3.5 Build Per-Skill Package

Each skill generation agent receives variables that map directly to the agent prompt template:

# Agent prompt variables (see agent-prompt.md for full template)
name: "libfuzzer"                              # {name} - skill name (lowercase)
type: "fuzzer"                                 # {type} - tool|fuzzer|technique|domain
pass: 1                                        # {pass} - 1=content only, 2=cross-refs only
handbook_path: "/path/to/testing-handbook"     # {handbook_path} - absolute path
section_path: "fuzzing/c-cpp/10-libfuzzer"     # {section_path} - relative to content/docs/
output_dir: "/path/to/skills/plugins/testing-handbook-skills/skills"  # {output_dir}
template_path: "skills/testing-handbook-generator/templates/fuzzer-skill.md"  # {template_path}
related_sections: "fuzzing/techniques/01-writing-harnesses, fuzzing/techniques/03-asan"
# ^ Use comma-separated list OR empty string "" if no related sections

# Metadata (for orchestrator tracking, not passed to agent)
metadata:
  title: "libFuzzer"                           # From frontmatter
  summary: "Coverage-guided fuzzer..."         # From frontmatter
  has_resources: true                          # Has 99-resources.md
  estimated_lines: 350                         # Approximate output size

Variable mapping to agent prompt:

Building the package:

# Construct output_dir from current location
output_dir="$(cd .. && pwd)"  # Parent of testing-handbook-generator

# Construct template_path from type
template_path="skills/testing-handbook-generator/templates/${type}-skill.md"

# Format related sections (empty string if none)
if [ ${#related_sections[@]} -eq 0 ]; then
  related_sections=""
else
  related_sections=$(IFS=', '; echo "${related_sections[*]}")
fi

3.6 Pre-Generation Validation

Before launching generation agents, verify:

For each skill candidate:
├─ Primary content exists and is non-empty
├─ Frontmatter has title and summary
├─ At least one code block present (for tool/fuzzer types)
├─ No unresolved Hugo shortcodes in source content
├─ Related sections (if any) are accessible
└─ Template file exists for skill type

If validation fails:

• Log the specific failure
• Move candidate to "Skipped Sections" with reason
• Continue with remaining candidates

3.7 Launch Generation

Trigger conditions:

• User has approved plan (explicit confirmation)
• All source paths verifi