Agent Observability

At a glance

• Outcome: Captures six coding-agent log formats in one queryable system. Reports failures, token use and recurring anti-patterns across six harnesses.
• Status: Runs daily on the author’s agent sessions as open-source software. As of September 11, 2026, public main matches its August 27, 2026 commit.
• Role: Solo. I own problem selection, architecture, acceptance criteria, integration, review and debugging; implementation is AI-assisted and reviewed.
• Stack & libraries: Go 1.25, SQLite, CLI. One database per harness.
• Source: Public, MIT-licensed: github.com/Griffinhale/agent-observability, with CI running the full suite.
• Validation: Runs 260 Go tests across 56 files in CI on Linux, macOS and Windows, covering parsers, pipeline stages, archive atomicity and swap rollback. Uses a real-session fixture corpus and a 2026-06-18 reconciliation audit across all six parsers on real installs: about 6,000 sessions and 118,000 tool calls reconciled silver counts with gold aggregates for every harness. The audit records one dated result rather than a rerunning gate.
• Limitations: restore writes nontransactionally and can leave a half-written directory, while swap is all-or-nothing; credential stashes keep hashes without bytes, preventing rollback after forced overwrite. Windows cannot preserve permission bits, restored credentials are readable by any local user, and captured archives record 0666 for files that were never protected; the tool cannot fix this. English text heuristics affect redirect counts, anti-pattern counts and session scores; gemini and antigravity have fidelity gaps tracked in public issues, only claude and codex use live-install tests, healthy harnesses can cross audit tripwires, so tripwires prompt review rather than decide health, and no tagged release exists despite the release workflow, requiring go install or a cloned build.

Capture model

Captures transcripts, caches and history so heavy live directories can be trimmed without losing audit data. Stores each artifact with its checksum, harness-relative path and capture run. Links every normalized interaction back to its artifact, letting reports trace numbers to transcript bytes and rebuild scoring without changing evidence.

Classifies files through per-harness rules as config, conversation, secret or heavy. Encrypts secrets or records tombstones, stores reconstructable heavy files as manifest-only, and swaps only config. Captures unmatched paths under the default tier. Adding a harness requires a rule table, parser, registry entry and parser-lookup case; classification does not parse formats.

Data pipeline

Stores raw captures unchanged in bronze, normalizes six formats into silver, and derives failure, token and anti-pattern analysis in gold. Recomputes gold projections from cleared tables on every run. Reads structural error fields where available, calculates token counts arithmetically and matches English prompt keywords for anti-patterns; the schema does not distinguish their different strengths.

Keeps one SQLite database per harness to isolate parsers and migrations. Queries raw captures, normalized sessions and derived statistics together inside each database. Builds cross-harness audit and dashboard views through the CLI reading every database in the data directory.

Operations

• Captures, archives and restores portable sessions to working state.
• Normalizes six parsers so analysis code does not depend on the originating tool.
• Swaps and rolls back configuration with diffs for comparable experiments.
• Reports failure, token and anti-pattern statistics from the derived layer.

Failure classification

Replaced failure-word scans that misclassified successful reads containing “error” and ordinary shell output containing “failed.” Parsers now persist nullable verdicts from harness error fields, use text heuristics only without structural signals and trust present exit codes.

Correlates delayed Claude tool results through interaction and tool-call indices. Avoids pointers into the growing interaction slice, which appends can reallocate and invalidate.

Write safety

Prompts before both destructive writes, backs up replacements into one stash ring and verifies recorded SHA256 values before writing. Rejects archives with paths outside the target or newer writer versions. Accepts prompts through --yes; refuses nonterminal writes without it.

Builds swap replacements separately and commits OS renames that restore the previous tree after a final failure. Writes restore files individually, reports its stopping point and provides no transaction.

Fixed rollback names that previously resolved only against named slots, leaving every on-disk auto-stash listed by swap list but unreachable. The forced-restore rollback command had failed with slot not found. The fix also prevented ring-cap eviction from deleting the applied stash and prevented second-resolution timestamps from creating duplicate names during forced-restore rollback.