Configurable Markdown Source of Truth

Overview

Enable markdown files to be the configurable source of truth for entity data, inverting the current JSONL-first architecture. This allows teams who prefer human-readable, wiki-style documentation to treat markdown files as authoritative.

Current Architecture

Markdown Files (.sudocode/specs/*.md)
    ↕ (syncs via watcher, timestamp-based conflict resolution)
JSONL Files (specs.jsonl, issues.jsonl) ← CURRENT SOURCE OF TRUTH
    ↕ (import/export)
SQLite Cache (cache.db) ← QUERY ENGINE (gitignored)

Proposed Architecture (Configurable)

Mode: "markdown" (new)                    Mode: "jsonl" (current default)
─────────────────────────                 ─────────────────────────────────
Markdown Files ← SOURCE OF TRUTH          Markdown Files (optional)
    ↓ (derived)                               ↕ (syncs)
JSONL Files (cache/export)                JSONL Files ← SOURCE OF TRUTH
    ↓                                         ↓
SQLite Cache                              SQLite Cache

---

Implementation Plan

Phase 0: Config Split (Prerequisite)

Split config.json into tracked and local files. Currently config.json is gitignored, but sourceOfTruth needs to be shared across the team.

0.1 New config file structure

.sudocode/
├── config.json        # Git-tracked, shared project settings
├── config.local.json  # Gitignored, machine-specific settings
└── ...

0.2 Update types (types/src/index.d.ts)

/**
 * Storage mode determines the source of truth for entity data
 */

/**
 * Shared project configuration (git-tracked: .sudocode/config.json)
 */

  /** Source of truth for entity data (default: "jsonl") */
  sourceOfTruth?: StorageMode;
  /** Integration configurations (shared across team) */
  integrations?: IntegrationsConfig;
}

/**
 * Local machine configuration (gitignored: .sudocode/config.local.json)
 */

  /** Worktree configuration (machine-specific paths) */
  worktree?: WorktreeConfig;
  /** Editor configuration (personal preference) */
  editor?: EditorConfig;
  /** Voice configuration (personal preference) */
  voice?: VoiceSettingsConfig;
}

/**
 * Merged configuration (ProjectConfig + LocalConfig)
 */

  /** @deprecated Use ProjectConfig fields directly */
  version?: string;
}

0.3 Update config.ts (cli/src/config.ts)

const PROJECT_CONFIG_FILE = "config.json";
const LOCAL_CONFIG_FILE = "config.local.json";

/**
 * Read project config (git-tracked)
 */
function readProjectConfig(outputDir: string): ProjectConfig {
  const configPath = path.join(outputDir, PROJECT_CONFIG_FILE);
  if (!fs.existsSync(configPath)) {
    return {};
  }
  return JSON.parse(fs.readFileSync(configPath, "utf8"));
}

/**
 * Read local config (gitignored)
 */
function readLocalConfig(outputDir: string): LocalConfig {
  const configPath = path.join(outputDir, LOCAL_CONFIG_FILE);
  if (!fs.existsSync(configPath)) {
    return {};
  }
  return JSON.parse(fs.readFileSync(configPath, "utf8"));
}

/**
 * Get merged config (project + local)
 */

  const project = readProjectConfig(outputDir);
  const local = readLocalConfig(outputDir);
  return { ...project, ...local };
}

/**
 * Update project config (git-tracked)
 */

  outputDir: string,
  updates: Partial
): void {
  const config = readProjectConfig(outputDir);
  Object.assign(config, updates);
  const configPath = path.join(outputDir, PROJECT_CONFIG_FILE);
  fs.writeFileSync(configPath, JSON.stringify(config, null, 2), "utf8");
}

/**
 * Update local config (gitignored)
 */

  outputDir: string,
  updates: Partial
): void {
  const config = readLocalConfig(outputDir);
  Object.assign(config, updates);
  const configPath = path.join(outputDir, LOCAL_CONFIG_FILE);
  fs.writeFileSync(configPath, JSON.stringify(config, null, 2), "utf8");
}

/**
 * Check if markdown is the source of truth
 */

  return config.sourceOfTruth === "markdown";
}

0.4 Update .gitignore (.sudocode/.gitignore)

  cache.db*
  issues/
  specs/
  worktrees/
- config.json
+ config.local.json
  merge-driver.log
  deploy-config.json
  spawn-config.json

0.5 Migration for existing projects

When loading config, migrate old config.json to new split:

function migrateConfigIfNeeded(outputDir: string): void {
  const oldConfigPath = path.join(outputDir, "config.json");
  const localConfigPath = path.join(outputDir, LOCAL_CONFIG_FILE);

  if (fs.existsSync(oldConfigPath) && !fs.existsSync(localConfigPath)) {
    const oldConfig = JSON.parse(fs.readFileSync(oldConfigPath, "utf8"));

    // Extract local-only fields
    const localConfig: LocalConfig = {};
    if (oldConfig.worktree) localConfig.worktree = oldConfig.worktree;
    if (oldConfig.editor) localConfig.editor = oldConfig.editor;
    if (oldConfig.voice) localConfig.voice = oldConfig.voice;

    // Extract project fields
    const projectConfig: ProjectConfig = {};
    if (oldConfig.integrations) projectConfig.integrations = oldConfig.integrations;
    // sourceOfTruth would be new, not in old config

    // Write split configs
    if (Object.keys(localConfig).length > 0) {
      fs.writeFileSync(localConfigPath, JSON.stringify(localConfig, null, 2));
    }
    fs.writeFileSync(oldConfigPath, JSON.stringify(projectConfig, null, 2));
  }
}

0.6 Files to modify

---

Phase 1: Watcher Logic Changes

2.1 File Deletion Handling (cli/src/watcher.ts:470-477)

Current behavior:

if (event === "unlink") {
  // File was deleted - DB/JSONL is source of truth, so we don't delete entities
  onLog(`[watch] Markdown file deleted: ${relPath} (DB/JSONL is source of truth)`);
}

New behavior:

if (event === "unlink") {
  const config = getConfig(baseDir);
  if (isMarkdownFirst(config)) {
    // Markdown is source of truth - delete entity from DB
    const entityId = getEntityIdFromPath(filePath, db, entityType);
    if (entityId) {
      if (entityType === "spec") {
        deleteSpec(db, entityId);
      } else {
        deleteIssue(db, entityId);
      }
      onLog(`[watch] Deleted ${entityType} ${entityId} (markdown file removed)`);
      await exportToJSONL(db, { outputDir: baseDir });
    }
  } else {
    // JSONL is source of truth - ignore file deletion
    onLog(`[watch] Markdown file deleted: ${relPath} (JSONL is source of truth)`);
  }
}

2.2 Orphaned File Handling (cli/src/watcher.ts:540-554)

Current behavior: Deletes markdown files without DB entries

New behavior:

if (syncDirection === "orphaned") {
  const config = getConfig(baseDir);
  if (isMarkdownFirst(config)) {
    // Markdown is source of truth - CREATE entity from markdown
    const syncResult = await syncMarkdownToJSONL(db, baseDir, filePath, {
      autoInitialize: true,  // Generate ID, set defaults
      entityType,
    });
    if (syncResult.success) {
      onLog(`[watch] Created ${entityType} from markdown: ${relPath}`);
      await exportToJSONL(db, { outputDir: baseDir });
    }
  } else {
    // JSONL is source of truth - delete orphaned file
    fs.unlinkSync(filePath);
    onLog(`[watch] Deleted orphaned file: ${relPath}`);
  }
}

2.3 Sync Direction Logic (cli/src/watcher.ts:517-532)

Current behavior: Timestamp comparison, DB wins in ties

New behavior:

const config = getConfig(baseDir);
if (isMarkdownFirst(config)) {
  // Markdown is source of truth - always sync markdown → DB
  // (unless content matches, to prevent oscillation)
  syncDirection = "markdown-to-db";
} else {
  // JSONL/DB is source of truth - use timestamp comparison
  if (dbTime > fileTime) {
    syncDirection = "db-to-markdown";
  } else {
    syncDirection = "markdown-to-db";
  }
}

2.4 Startup Orphan Cleanup (cli/src/watcher.ts:928-994)

Current behavior: Deletes all markdown files without DB entries on startup

New behavior:

// For each markdown file without DB entry:
const config = getConfig(baseDir);
if (isMarkdownFirst(config)) {
  // Create entity from markdown file
  await syncMarkdownToJSONL(db, baseDir, filePath, { autoInitialize: true });
} else {
  // Delete orphaned file (current behavior)
  fs.unlinkSync(filePath);
}

---

Phase 2: Sync Commands

3.1 Update determineSyncDirection() (cli/src/cli/sync-commands.ts:416-538)

Current behavior: Prefers database in conflicts (line 531)

New behavior:

function determineSyncDirection(ctx: CommandContext): {
  direction: "to-markdown" | "from-markdown" | "no-sync";
  reason: string;
} {
  const config = getConfig(ctx.outputDir);

  if (isMarkdownFirst(config)) {
    return { direction: "from-markdown", reason: "Markdown is source of truth" };
  } else {
    // Use existing timestamp-based logic for jsonl mode
    return existingTimestampLogic();
  }
}

---

Phase 3: Entity Operations

4.1 Spec Creation (cli/src/cli/spec-commands.ts)

Current flow: Generate ID → Create in SQLite → Export JSONL → Sync markdown

New flow (markdown-first):

if (isMarkdownFirst(config)) {
  // 1. Generate ID
  // 2. Write markdown file with frontmatter
  // 3. Sync markdown → DB (watcher or explicit)
  // 4. Export JSONL (always exported)
} else {
  // Current flow
}

4.2 Spec Deletion (cli/src/cli/spec-commands.ts:415-448)

Current flow: Delete from DB → Delete markdown file → Export JSONL

New flow (markdown-first):

if (isMarkdownFirst(config)) {
  // 1. Delete markdown file
  // 2. Watcher triggers DB deletion
  // 3. Export JSONL (always exported)
} else {
  // Current flow
}

4.3 Issue Operations (cli/src/cli/issue-commands.ts)

Same pattern as specs.

---

Phase 4: Import/Export Changes

5.1 Add rebuildFromMarkdown() (cli/src/import.ts)

New function to rebuild JSONL + DB from markdown files:

/**
 * Rebuild database and JSONL from markdown files (markdown-first mode)
 */

  db: Database,
  outputDir: string,
  options?: { entityType?: "spec" | "issue" }
): Promise<{ specs: number; issues: number }> {
  const specsDir = path.join(outputDir, "specs");
  const issuesDir = path.join(outputDir, "issues");

  let specCount = 0;
  let issueCount = 0;

  // Process spec markdown files
  if (!options?.entityType || options.entityType === "spec") {
    for (const file of glob.sync("**/*.md", { cwd: specsDir })) {
      await syncMarkdownToJSONL(db, outputDir, path.join(specsDir, file), {
        autoInitialize: true,
        entityType: "spec",
      });
      specCount++;
    }
  }

  // Process issue markdown files
  if (!options?.entityType || options.entityType === "issue") {
    for (const file of glob.sync("**/*.md", { cwd: issuesDir })) {
      await syncMarkdownToJSONL(db, outputDir, path.join(issuesDir, file), {
        autoInitialize: true,
        entityType: "issue",
      });
      issueCount++;
    }
  }

  // Export to JSONL
  await exportToJSONL(db, { outputDir });

  return { specs: specCount, issues: issueCount };
}

5.2 Update JSONL import (cli/src/import.ts)

Add config check before importing:

  const config = getConfig(outputDir);

  // In markdown-first mode, JSONL is derived - skip import or warn
  if (isMarkdownFirst(config)) {
    console.warn("JSONL import skipped: markdown is source of truth");
    console.warn("Run 'sudocode sync --from-markdown' to rebuild from markdown files");
    return { specs: 0, issues: 0, skipped: true };
  }

  // Existing import logic...
}

---

Phase 5: CLI Commands

6.1 New sync subcommand

# Rebuild DB + JSONL from markdown (markdown-first mode)
sudocode sync --from-markdown

# Rebuild markdown from JSONL (jsonl-first mode)
sudocode sync --to-markdown

# Auto-detect based on config
sudocode sync --auto

6.2 Config command for storage mode

# Set source of truth
sudocode config set sourceOfTruth markdown

# View current setting
sudocode config get sourceOfTruth

---

Phase 6: Server Changes

7.1 Export service (server/src/services/export.ts)

Add config awareness:

  const config = getConfig(outputDir);

  // Always export JSONL (for git tracking, regardless of mode)
  await exportToJSONL(db, { outputDir });

  if (isMarkdownFirst(config)) {
    // In markdown-first mode, also update markdown files from DB
    // (for changes made via API that didn't go through markdown)
    await syncDBToMarkdown(db, outputDir);
  }
}

---

Phase 7: Documentation Updates

8.1 Files to update

---

Phase 8: Test Updates

9.1 Test files requiring changes

8.2 New test scenarios

describe("markdown-first mode", () => {
  it("creates entity when new markdown file is added");
  it("deletes entity when markdown file is removed");
  it("markdown changes always sync to DB");
  it("JSONL is regenerated after markdown changes");
  it("orphaned markdown files create new entities");
  it("git pull with JSONL changes does not overr