Give Your AI Agents a Second Brain

Your AI agents can write code, plan projects, and solve difficult problems. Then the session ends — and the next one starts without the decisions you already paid to reach.

Librarian is a self-hosted Second Brain for AI coding agents. It turns completed terminal sessions into compact, searchable memories. Later, a connected agent can ask “have we dealt with this before?” and retrieve the relevant decision, fix, or dead-end instead of rebuilding context from zero.

This article shows the production system behind the product: what it remembers, how it works, where it failed, and what I changed after running it every day.

Role	Solo architect & operator — I owned the problem, the design decisions, the production system, and its incidents. Implementation was AI-directed (I specified and reviewed; Claude Code wrote the code under my direction).
Type	Personal infrastructure turned into a self-hosted Starter Kit
My setup	Node.js · PostgreSQL + pgvector · Gemini Flash · Vertex AI embeddings · MCP · Raspberry Pi 500 · Tailscale. The Starter Kit can run anywhere you have Docker; the Pi is my example, not a requirement.
Status	In production since April 2026; incident-hardened release since May 2026. Used daily as the memory layer for the several Claude Code agents I run.
Artifacts	Architecture doc · 11-entry Architectural Decision Log (ADRs) · operations runbook · incident log — all maintained alongside the code.

1. The problem: every session starts from zero

AI agents are stateless between sessions. Every new session starts blind: the agent has no idea what was decided yesterday, which approach already failed last week, or what state a long-running project is in.

Claude Code auto-writes a full transcript of every session to disk — but a raw transcript is the wrong artifact to remember from:

• It is too long to reload into a context window.
• It is too noisy to search — ninety percent is mechanics; the value is a few decisions buried inside.
• Document-style RAG answers “what does this document say”, which is a different question from “what did I already do about this.”

The cost of no memory is concrete: the agent repeats mistakes, re-litigates settled decisions, and loses continuity on anything spanning more than one sitting. I was paying — in time and tokens — to re-explain context I had already established.

The task I set myself: capture the signal from each session, make it semantically searchable, and expose it to compatible agents through a clean interface — without turning raw transcripts into permanent memory or requiring a frontier-model bill.

For the person using the agent, the outcome is simple:

• Less time repeating project context.
• Fewer rediscovered dead ends.
• Decisions and fixes remain available after the chat window is gone.
• The memory runs on infrastructure you control.

2. From my production system to a Starter Kit

I built Librarian for my own work first. I defined the behaviour and architecture, directed the AI-assisted implementation, reviewed the output, and operate the production system. The useful part is not a prompt or a screenshot; it is the complete memory loop and the operational decisions behind it.

• Write: completed supported terminal sessions become factual, structured digests.
• Remember now: an agent can save one deliberate memory when you say “remember this.”
• Recall: compatible agents search and retrieve memories through four real MCP tools.
• Verify first: synthetic fixtures prove the loop before you connect real session data.

The commercial Starter Kit packages that core into a self-hosted installation for Docker, with source code, migrations, setup instructions, security boundaries, troubleshooting, and written support.

3. The core idea — remember the signal, not the transcript

The design thesis, and the thing that separates Librarian from a naive “log everything and keyword-search it” store:

An LLM sits on the write path and an LLM sits on the read path — and that is deliberate.

• On write (input): raw transcripts are never stored as memory. An LLM (Gemini Flash) reads the session and produces a compact, structured digest — what we worked on / what problem / how it was solved / what’s left open — plus tags, a domain, an importance score, and a one-sentence summary. Signal extracted, noise discarded: recording everything indiscriminately produces a haystack, not a memory.
• On read (output): recall is not auto-injected into every prompt. It is a tool the consuming agent chooses to call, and the agent applies its own judgment to what comes back — deciding whether a recalled memory is relevant, current, and worth acting on.

The bet is that selectivity at both ends beats a dumb pipe in either direction — dump-everything on write, or blind-inject on read. The smart part is what gets remembered, and what gets recalled and used. I validate this by spot-checking recall against sessions I remember; an automated recall-quality harness (a golden query set) is the top open item — I do not yet measure precision numerically, and I say so plainly rather than assert it.

Everything below is in service of this thesis.

4. Architecture

Two independent services, cleanly split into a writer and a reader — a system can capture memory and serve memory without the two paths interfering.

My production instance mirrors session logs from a Mac to a Raspberry Pi 500, where a harvester processes them. That is one deployment example. The Starter Kit uses Docker Compose and can run on macOS, Linux, or a loopback-only VPS.

Two-layer storage. One memories row per session holds the full human-readable digest; many indexed_chunks rows hold one embedding per bullet, each linked back to its parent. Retrieval ranks on the fine-grained chunks, then follows the link to the full digest only when needed — a cheap two-step over one expensive blob.

Three-level scope isolation. project (coarse) → domain (six fixed buckets: technical, business, reflections, core, content, private) → tags (open-ended). An agent can narrow before it searches, so recall is targeted rather than fuzzy — and private-domain memory stays isolated from shared agents.

MCP as the interface. Exposing recall through the Model Context Protocol means any MCP-compatible agent gets memory with zero custom glue — the same integration shape the agents already use for other tools.

Security posture (stated, not hidden). The recall endpoint has no application-level auth: the Tailscale network boundary is the perimeter, which is the right trade-off for single-operator use. Token-signing would be a prerequisite before any multi-user exposure — a deliberate, documented decision, not an oversight.

5. Selected engineering decisions

The full log has 11 entries; these four best show the reasoning style. Each was written down at the time with context, decision, and consequences.

Precision over recall — chunk by structure, not by sliding window. The industry-default RAG chunking is a ~500-character sliding window. Digests already have explicit structure (named sections, numbered bullets), so I chunk one bullet = one chunk instead. One concept per embedding raises ranking precision for narrow queries; a sliding window would have averaged unrelated topics into one vector and blurred recall. (Trade-off accepted: chunks vary in size and the chunker is digest-specific, not a general splitter.)

Model the memory, not a graph — an explicit non-goal. There is constant temptation to bolt on the latest graph-RAG paper (LightRAG, Mem0, etc.). I wrote an ADR drawing the line: those systems optimise recall over a static document corpus; Librarian optimises precision over personal session transcripts, where the priority is not hallucinating a decision I never made. The rule that came out of it — borrow retrieval-side mechanics (rerankers, query rewrite), never the storage model wholesale — turns every future “should we adopt X?” from a half-day comparison into a one-line check against a written non-goal.

Right-sized model choice. Digest extraction runs on Gemini Flash on the free tier, not a frontier model. Summarising a session doesn’t need frontier reasoning; the free tier covers extraction, and the only metered cost is Vertex AI embeddings (billed per character, cents/month at this volume). Right tool for the job, not the most expensive one — a cost decision I can justify.

Independence over coupling. A neighbouring MCP service had corrupted source on disk (see §7 lineage). Rather than block on repairing it, I made the writer talk directly to PostgreSQL and gave Librarian its own recall server. The system now lives and dies on its own code — no shared component can take it down.

6. A design pivot under real signal

The original MVP was a “Correction Hunter” — extract every “don’t do X, do Y” moment from a session and store it as a rule. First smoke tests killed it: the LLM over-generalised a single in-context remark into a blanket rule, and the same remark phrased differently produced duplicate rules that hashing couldn’t dedupe.

I flipped the output format mid-build: one factual digest per session instead of N prescriptive rules. Digests are fact-shaped (“what happened”), which is a far smaller hallucination surface than prescriptive rules (“what should always be done”), and one-record-per-session made idempotency trivial (hash(session_id) — one row by construction).

The lesson I take to interviews: I changed the core abstraction because the evidence said to, not because the original plan was sunk cost.

7. Results

• In production since 16 April 2026, used daily as the live memory layer for the several agents I run — this is not a demo. Roughly three months of continuous operation, interrupted only by the incident below.
• 550+ sessions digested · several thousand searchable digest chunks (5–15 per session), embedded and queried by cosine similarity across six domains — dominated by technical work with a steady tail of business and reflections. (These counts are the genuinely-digested-since-launch figures; they exclude ~4,150 pre-launch sessions marked skip and a separate legacy file-index corpus that recall filters out.)
• Marginal running cost — cents/month. Free-tier extraction model, self-hosted Postgres on a Raspberry Pi; the only metered line is per-character embeddings at low volume.
• In daily use it answers three recurring questions well enough that I rely on it: did we already solve this? · what was the state of this project? · what happened last time we touched this system?

A production incident I own. A deployment stripped the executable bit off a service wrapper; after the next reboot the harvester silently failed to start — a six-day gap in memory capture before I caught it. The fix was not a one-off patch but a set of durable guards I designed and documented as invariants:

• the service now re-asserts the executable bit on every start (ExecStartPre=chmod +x), so the failure mode cannot recur;
• restart-rate limits were corrected so a genuinely failing service actually enters the failed state instead of restart-looping forever;
• an OnFailure hook now fires a Telegram alert — so the next silent failure is loud;
• enable-linger guarantees user services come up on boot without a login.

Diagnosing a silent, delayed-onset production failure and converting it into codified invariants — rather than a quiet hotfix — is the operational maturity these roles look for.

8. Tradeoffs & lessons (honest notes)

• Summarisation is lossy by design. The digest is a lens, not a mirror; the summariser’s quality is the memory’s quality. Mitigation is one call away (get_digest), but a weak model yields weak memory.
• Taxonomy needs discipline. Domains and tags only help if applied consistently; left unmanaged they drift into noise. I enforce normalisation on the write path and seed the vocabulary into the extraction prompt — belt and braces.
• Semantic recall can mislead. Cosine similarity surfaces things that sound related but aren’t. Recall is treated as a starting point for the agent’s judgment, not a verdict — which is exactly why the read path keeps an LLM in the loop (§3).
• Recency vs. relevance is a real tension. The most similar memory isn’t always the most current one; a stale digest can describe a state that no longer exists. So recall is context to verify against live reality, never truth to act on blindly.

9. What the Starter Kit gives you

What you get	Evidence behind it
A complete memory loop	Session harvest, explicit write_memory, semantic search, and full-digest retrieval
Self-hosted ownership	PostgreSQL + pgvector in Docker; your machine, accounts, and data
Agent-first installation	START_HERE.md and examples for supported Codex and Claude Code terminal sessions
Proof before private data	Synthetic fixtures and an automated end-to-end verification
Operational guidance	Setup, privacy, security, limitations, troubleshooting, and uninstall documentation
Commercial use	One buyer or legal entity may use and modify the kit for its own internal and commercial work

---

Built and operated by Slava Spitsyn. Source is private (personal infrastructure); this write-up describes the system without exposing secrets, credentials, or private content.

---

← All notes