Memory Architecture: QMD Migration + Artifact Simplification

Executive Summary

Migrate from the builtin Gemini-only memory search to QMD hybrid search (BM25 + vector + reranking), and simultaneously simplify the artifact system from a 5-file packaging pipeline to plain markdown files that QMD auto-discovers. Two phases, done in reverse order: simplify artifacts first (less risk, unblocks Phase 2), then swap the search backend.

Current State

What exists today

Memory recall path (what agents can search):
  MEMORY.md ─────────────────── always loaded in context
  memory/*.md (43 files, 404K) ─ indexed by builtin Gemini embeddings
  memory/YYYY-MM-DD.md ──────── today + yesterday auto-loaded

Artifact storage (NOT searchable):
  mission-control/artifacts/by-id/
    art_*/
      source.md          ← the actual content
      meta.json           ← packaging metadata
      preview.html        ← rendered HTML
      preview-card.svg    ← summary card image
      preview.pdf         ← PDF export

Workspace files (always loaded, not searched):
  AGENTS.md, SOUL.md, USER.md, TOOLS.md, IDENTITY.md, HEARTBEAT.md

Problems

1. Artifacts are invisible to recall. 8 artifact source files with real research and decisions. Agents can't find them via memory_search.
2. Artifact creation is fragile. Agents must run create-artifact.mjs to register. Raw writes produce orphaned files (happened today with the LinkedIn artifact).
3. 5 files per artifact is over-engineered. meta.json, preview.html, preview-card.svg, preview.pdf exist for a presentation layer that could render on-the-fly from source markdown.
4. Builtin search is vector-only. No keyword matching, so exact terms (project names, issue numbers) can miss. QMD adds BM25 for lexical precision.

Target State

Memory + Knowledge recall path (all searchable via QMD):
  MEMORY.md ─────────────────── always loaded in context
  memory/*.md ───────────────── daily notes, session logs
  artifacts/*.md ────────────── research, plans, specs, analysis
  (optional) references/*.md ── ingested external docs

MC presentation layer (renders from source):
  MC reads artifacts/*.md directly
  Extracts title from H1, summary from first paragraph
  Renders via shared markdown renderer
  No meta.json, no preview files, no registration step

Search backend:
  QMD (BM25 + vector + reranking, fully local)
  Auto-indexes all paths on 5-minute interval
  Zero API cost for search operations

Phase 1: Simplify Artifacts (do this first)

Why first: This is lower risk and produces the directory structure QMD will index. No point indexing the current mess of meta.json and preview files.

1.1 New artifact directory

Move artifact source files to a flat, simple location:

workspace/artifacts/
  voice-solutions-analysis.md
  agent-operating-protocol-v1.md
  linkedin-post-analysis-2026.md
  mc2-architecture.md
  ...

Rules:

• One markdown file per artifact. That's it.
• Filename is the slug (kebab-case, descriptive).
• No meta.json, no preview files, no registration JSON.
• Title comes from the first H1 in the file.
• Summary comes from the first paragraph (or an explicit > summary blockquote).
• Type/owner/tags can live in a YAML frontmatter block if needed, but are optional.

1.2 MC auto-discovery

Rewrite the MC artifact list API to scan workspace/artifacts/*.md instead of reading data-v2/artifacts.json:

• Glob *.md files from the artifacts directory
• Parse frontmatter (if present) for optional metadata
• Extract H1 as title, first paragraph as summary
• Compute word count and read time from content
• Render via the existing shared markdown renderer
• No packaging step, no registration, no create-artifact.mjs

1.3 Migration

• Copy each source.md from mission-control/artifacts/by-id/art_*/ to workspace/artifacts/ with clean filenames
• Verify MC displays them correctly
• Remove old artifacts/by-id/ directory and data-v2/artifacts.json
• Remove create-artifact.mjs and seed-artifacts.mjs
• Update AGENTS.md: "To create an artifact, write a markdown file to workspace/artifacts/"

1.4 Agent workflow change

Before:

Agent writes source.md → runs create-artifact.mjs → MC sees it

After:

Agent writes workspace/artifacts/my-research.md → MC sees it

Definition of done (Phase 1)

• [ ] workspace/artifacts/ contains all current artifact content as plain .md files
• [ ] MC lists and renders artifacts from that directory (no JSON registry)
• [ ] Full-screen/export view works from the same source
• [ ] Creating a new artifact = writing a .md file (no script needed)
• [ ] Old packaging pipeline removed
• [ ] AGENTS.md updated with new artifact creation rule

Phase 2: QMD Search Backend

Why second: Now that artifacts are clean .md files in a known location, QMD can index them alongside memory files.

2.1 Fix QMD installation

Current state: QMD binary exists at /Users/vinny/.bun/bin/qmd but module is missing. Fix:

bun install -g @tobilu/qmd    # reinstall cleanly
qmd --version                  # verify

Prerequisites already met:

• Bun 1.3.10 installed
• SQLite 3.51.0 available
• 370GB free disk space
• Mac Mini ARM64

2.2 Configure QMD as memory backend

{
  "memory": {
    "backend": "qmd",
    "citations": "auto",
    "qmd": {
      "includeDefaultMemory": true,
      "paths": [
        {
          "name": "artifacts",
          "path": "artifacts",
          "pattern": "**/*.md"
        }
      ],
      "update": {
        "interval": "5m",
        "debounceMs": 15000,
        "onBoot": true,
        "waitForBootSync": false
      },
      "limits": {
        "maxResults": 8,
        "maxSnippetChars": 700,
        "timeoutMs": 4000
      },
      "scope": {
        "default": "deny",
        "rules": [
          { "action": "allow", "match": { "chatType": "direct" } },
          { "action": "allow", "match": { "keyPrefix": "discord:channel:" } }
        ]
      }
    }
  }
}

This indexes:

• MEMORY.md + memory/**/*.md (via includeDefaultMemory)
• artifacts/**/*.md (via paths)

2.3 Verify search quality

Test queries that should surface artifacts:

• "voice solutions discord" → should find voice-solutions-analysis.md
• "agent operating protocol" → should find agent-operating-protocol-v1.md
• "LinkedIn post performance" → should find linkedin-post-analysis-2026.md
• "mission control architecture" → should find mc2-architecture.md

Test queries that should still find memory:

• "Pete's timezone" → MEMORY.md
• "David's model config" → daily notes
• "job search pipeline" → memory files

2.4 Remove builtin search config

Once QMD is verified:

• Remove agents.defaults.memorySearch Gemini config (QMD handles embeddings locally)
• Or keep Gemini as fallback if QMD fails (OpenClaw auto-falls-back)

Definition of done (Phase 2)

• [ ] QMD installed and running (qmd --version works)
• [ ] Config switched to QMD backend
• [ ] Artifacts appear in memory_search results
• [ ] Memory files still appear in memory_search results
• [ ] Search quality verified with test queries above
• [ ] No regression in daily agent operations

What changes for agents

Risks

Timeline estimate

• Phase 1 (artifact simplification): 1 David session, ~45 min
• Phase 2 (QMD migration): 1 session, ~30 min for install + config + verification
• Total: ~2 hours of agent work, not counting Pete review time