I built an AI agent that knows me

Every time you open a fresh chat with an LLM, it forgets who you are. Your role, your projects, the decision you made last week, the way you like answers framed — gone. You re-explain yourself, or you don’t and the answers stay generic. The model is brilliant and amnesiac at the same time.

So I set up a personal agent that doesn’t start from zero. I run NousResearch’s open-source Hermes agent on my own VPS, with a patched gbrain fork as its memory and a pipeline I wrote to turn my past sessions into sourced notes, and it talks to me over Telegram and the terminal, runs scheduled jobs, and, most importantly, remembers. Wiring that up taught me that the interesting engineering problem in “AI agents” right now isn’t the model. It’s the memory.

The naive version, and why it breaks

The obvious approach is: take everything about yourself and stuff it into the system prompt. Your bio, your projects, your preferences — paste it all in.

This falls apart fast. Context windows are finite and attention isn’t free: a giant wall of text makes the model worse, not better, because the signal drowns. Worse, a raw dump has no provenance — the agent can’t tell what’s a hard fact (“I live in X”), what’s a stale note from months ago, and what was a one-off. And it doesn’t grow: tomorrow you learn something new and there’s no clean place to put it.

Dumping your life into a prompt isn’t a memory system. It’s a junk drawer.

Treating my own history as a dataset

The shift that made it work: treat the memory layer as the actual product, and treat my own past as a dataset to mine.

I had years of my own conversations and notes sitting in exports and logs. I wrote a small extraction script to pull those past sessions into clean transcripts, then distilled them — not into one big file, but into many small ones. One fact per file, each with a little metadata: what kind of fact it is (who I am, a preference, a project, a reference), a one-line description used to decide relevance, and links to related facts. An index lists them all and loads at the start of every session; the full files load only when relevant.

Going forward, the agent captures new signal the same way: when I say something that’s actually a durable fact — a decision, a preference, a project constraint — it writes it down as its own small, linked, sourced note. Every fact carries a pointer back to where it came from. An unlinked, unsourced fact is a liability, not memory.

I borrowed this discipline partly from studying open “second brain” systems for agents: file by what something is, not what format it arrived in; back-link every entity; cite every fact; and don’t create a page for a one-off mention. Those four rules do more for answer quality than any clever retrieval trick.

The tradeoffs I’d call out

• Files over a vector database — at first. For a personal-scale brain, plain markdown files with a loaded index beat a vector store: zero infra, trivially inspectable, diff-able in git, and I can read my own memory. Vectors come later, when volume demands it — mine now runs a hybrid, with local embeddings doing semantic recall on top of the files.
• Curate, don’t accumulate. The temptation is to capture everything. But a memory that captures everything is as useless as one that captures nothing. The value is in the gate — what’s notable enough to keep.
• Own the substrate. I run this on my own VPS, not inside someone else’s assistant. My profile, my notes, my history — that’s exactly the data I don’t want to hand to a third party. Self-hosting is a feature, not a chore.

What it actually does now

The result is an agent that already knows my context when I message it, recalls decisions across days and channels, runs scheduled briefings, and compounds: every week it knows a little more, with receipts. It’s not magic — it’s a knowledge base with good filing discipline and a model on top.

The lesson, if you’re building with AI

The model is the commodity now. The differentiator is the memory and context layer around it — how you capture, curate, and cite what the system knows. If you’re putting AI into a product, the question that matters isn’t “which model?” It’s “what does this system remember, how does it know it’s true, and who owns that data?”

That’s the part I find worth building. More soon — next up: what I found when I had the agent analyze its own past sessions.