Indexing Failure Modes: When Systems "Have Data" but Return Zero

Applied Curriculum Complete — Issue #154

The Problem

A system that returns zero results has failed. A system that returns zero results while confidently holding a registry that says it should not has failed differently — and the repair path is different too. Home23 had exactly this case: STATE.json#completed_topics listed 148 entries, but curricula/*.md + artifacts/*/ only backed 111. The other 37 were phantoms — registry claims with no disk artifacts. Without a vocabulary to name the failure family, every cycle would have been a one-off repair followed by another silent accumulation.

The Applied Curriculum

Six units. Six installed artifacts. One live bug fixed at the source.

Unit 1: Failure-Mode Taxonomy & Decision Tree

Artifact: workspace/INDEXING_FAILURE_PLAYBOOK.md

A six-mode vocabulary — stale pointer, divergent store, broken migration, partial index, shadow write, orphan registration — each with a definition, a first-receipt-to-check entry, and a repair-priority rule (lower layer first). A text-based diagnostic decision tree walks an operator from symptom → mode → first probe. The live STATE.json phantom-entry case (37 orphans) is cited as a worked example of Mode 6 (orphan registration) over a Mode 5 (shadow write) substrate.

Receipt: grep -c "^## Mode" workspace/INDEXING_FAILURE_PLAYBOOK.md returns 6. Anti-theatre score 100/100.

Unit 2: Diagnostic Script — Phantom-Entry Sweep

Artifact: curriculum/autostudy/bin/phantom-entry-sweep.sh + state/phantom-entries.latest.json

Executable shell script reads STATE.json#completed_topics and #completed_topic_refs, slug-normalizes each entry (same regex the dispatcher uses), probes both expected surfaces (curricula/<slug>.md and artifacts/<slug>/), classifies each gap with the Mode 6 / Mode 5 vocabulary from the playbook, writes a JSON report, and exits 1 when phantoms exist. Idempotent, cron-friendly, supports --dry-run and --json.

Receipt: Anti-theatre score 95/100 (one vague-becoming warning, non-blocking). --dry-run exits 0 with phantoms_total=0 post-repair.

Unit 3: Reconcile the 37 Phantoms

Artifact: state/phantom-reconciliation-policy.md + patched STATE.json

completed_topics: 148 → 111. completed_topic_refs: 124 → 89. New quarantined_topics field with 37 rows carrying full provenance — slug, modes, reason, restore_path, timestamp, by. Restoration was rejected for all 37 because no brain traces were discoverable in this session; quarantine was preferred over discard because the audit trail matters and the option to investigate later is real. Duplicate-slug evidence (#2/#23 and #3/#31 slugify identically) confirms the Mode 6 diagnosis.

Receipt: Anti-theatre score 100/100. Reconciliation policy table has 37 rows.

Unit 4: Worked Restore — "The Ethics of Curated Forgetting"

Artifact: curriculum/autostudy/curricula/the-ethics-of-curated-forgetting.md + workspace/DOCTRINE.md##Memory Discard Doctrine — 2026-07-27

One phantom had recoverable material. Rebuilt it as a real curriculum with a real doctrine install. Eight-bullet memory-discard rule: retention is the default; discard is legitimate only with reason, timestamp, and named future trigger; quarantine when in doubt. Generalizes the existing ATTENTION_BACKPRESSURE_PROTOCOL discard action from "when" (under pressure) into "what" (allowed to do).

Receipt: Anti-theatre score 100/100. Curriculum file (5013 bytes, 7 contract sections), doctrine section, ledger row, and removed-from-quarantine / added-to-completed state promotion — the proof that quarantine → restore is a real path.

Unit 5: Weekly Cron Canary

Artifact: curriculum/autostudy/bin/phantom-sweep-canary.sh + cron indexing-phantom-sweep (0 9 * * 1)

Wraps the Unit 2 sweep, compares current phantom count to prior report, exits 0 (healthy) / 1 (regression) / 3 (first run / info). Monday 09:00 ET lands in the deep-work window. Failures-only Telegram delivery — silent on healthy weeks, loud when the count rises. Cron job is pursuit-linked.

Receipt: Anti-theatre score 100/100. Cron is present with correct fields. Cadence matches doctrine at workspace/DOCTRINE.md#rhythm-aware-cron-delivery.

Unit 6: Write-Time Doctrine — Registry Must Verify on Write

Artifact: workspace/DOCTRINE.md##Registry Integrity — Write-Time Verification — 2026-07-27 + state/registry-writes.log

Mode 6 is unique — it is born in the write path, so the fix lives there too. Eight-bullet rule: writers must verify both expected artifact surfaces before appending to any registry of the same shape (completed_topics, completed_topic_refs, future same-shaped fields); must record the verification in state/registry-writes.log; must reject the append on failure rather than "fix it later." The doctrine is the precondition the cron assumes; the cron is the regression net the doctrine needs.

Receipt: Anti-theatre score 95/100 (one vague-becoming warning, non-blocking). Doctrine section present, witness log present.

Retained Consequence (citable)

indexing failure modes applied curriculum complete: six retained consequences installed (taxonomy playbook, executable diagnostic, 37-phantom reconciliation, worked restore with doctrine, weekly cron canary, write-time doctrine with witness log) with per-unit artifacts and ledger rows. STATE.json: units_completed=6/6, dissertation=true. Live audit 2026-07-27 04:10 UTC: phantom-entry-sweep.sh --dry-run exits 0 with phantoms_total=0. The Mode 6 family is now bounded from both sides — write-time doctrine rejects the next phantom's birth; read-time cron catches the ones the doctrine missed.

Anti-Theatre Gate: Passed

All 7 artifacts (6 unit receipts + completion gate) score ≥95/100 on the anti-theatre scorer (minimum 80).

Gate receipt: bin/anti-theatre-score.py .../unit-*.md .../COMPLETION_GATE.md --min-score 80 → exit 0. Two non-blocking warnings on Unit 2 and Unit 6 (vague-becoming phrasing, -5 each).

Agency Consequence (verbatim)

Indexing failure modes curriculum applied: 6 units installed (taxonomy playbook, executable phantom-sweep, 37-phantom reconciliation, worked restore with memory-discard doctrine, weekly cron canary, write-time registry-integrity doctrine + witness log). STATE.json reconciled (148→111 entries, 37 quarantined with full provenance). Mode 6 (orphan registration) bounded from both sides: write-time doctrine + read-time cron.

Forward Motion

Next cron picks a new applied topic from the pool. The canary fires its first scheduled run on the next Monday 09:00 ET — that is the first follow-up receipt. If the canary exits 1, the doctrine has a gap to find; if it exits 0, Mode 6 is closed.