Agentic memory can mean many different things, and it is not easy to figure out what a particular product does. In this post we will look at a few core attributes of memory which can help with classifying different solutions. We will also see why code is fundamentally different from other domains.
Two Dimensions
Every memory type can be classified using two dimensions.
Dimension 1: memory types
Let's borrow the terms from cognitive science, because they fit.
Working memory is what the agent holds while it is actively doing something. It is the context window plus a few affordances stacked on top (like a file an agent maintains). It is short-lived and scoped to a session. Orchestration lives here: what the agent is doing right now, what it just tried, what it plans to try next.
Episodic memory is the recording. It is the exact recollection of what happened in a piece of work: what options were considered, what got rejected, how the thing progressed from empty file to merged PR. Episodic memory is tied to a unit of work, a session or part of one. Think of it as the security-camera footage of a task, or the meeting minutes. It does not summarize. It shows.
Semantic memory is the distilled version. Facts, PRDs, design documents, "here is how our auth works." Semantic memory is either boiled down from episodic memory after the fact or written by hand up front. It is the decisions made, not the meeting minutes. Shorter, more portable, and lossy.
Most tools that say "memory" often mean semantic. It is the easiest to demo because a list of facts looks like knowledge.
Dimension 2: what the memory is for
The second axis is purpose.
Personalized memory is about you. What you want, how you like to work, which patterns you reach for. It can be episodic or semantic, but most tools implement it as semantic: a stored profile of preferences.
Institutional memory is the same idea scaled to a team. It is far harder to build, not for technical reasons but for human ones. Personal preferences are easy to capture because there is only one person to agree with. Team preferences require agreement, and teams rarely agree cleanly on how things "should" be done. The document that claims to speak for the team is usually a negotiated fiction.
Code memory is the odd one out. Instead of focusing on you or your team, it focuses on how the code itself was produced. Not "Victor likes tabs" but "this retry logic exists because the payment API times out under load, and here is the session where that was discovered and handled." It too can be episodic or semantic.
Lay the two dimensions on a grid and every memory product lands somewhere on it.
The map is not the territory
Different kinds of memory come with different degrees of trust, and they trade precision against abstraction.
Semantic memory is the most abstract and the most likely to be wrong. It is a human or a model's compression of reality, and compression discards the details that turn out to matter. Episodic memory is less abstract and less likely to be wrong, because it records what actually happened rather than someone's summary of it. But even episodic memory cannot be fully trusted, because it is still a recording of a past state, and the world has moved since.
Then there is the code itself. Docs rot. Code does not. Code never lies about what it does, because it is the thing that does it. It is the only artifact in the stack that is, by definition, current.
No matter how good your memory is, it is a map, and the map is not the territory. Which is why the most important capability is not better recall. It is materialization: the ability to pull down the actual state of any session, spin it up, and look at the ground truth.
Suppose your memory hands you a few facts about how an API in the backend works. The facts might even be mostly right. But the only way to know how that API behaves is to check out the repo and run the command that exercises it. Materialization gives you that.
It also gives you more. You can run speculative experiments, like changing a property on the backend to see what breaks. You can answer questions that were never stored in memory at all, like what happens when a third-party service goes down. A fact set can only answer the questions someone thought to record. A running system answers the questions you actually have.
A key point is that for a lot of coding, "doing by example" beats "consuming a set of distilled facts." Not always, and not as a full replacement for semantic memory, but reliably for consistency and for predictable questions. Show the agent the real code that solved a problem and let it replicate the shape, and you get a more faithful result than any bullet list of guidance would produce.
What this unlocks
Once you can find the right episode and materialize it, a set of workflows opens up that semantic memory simply cannot support.
- You can review a coworker's PR by materializing the session behind it and asking what options they considered and why the code landed the way it did, instead of reverse-engineering intent from a diff.
- You can establish best practices by finding the sessions that did a thing well and building new code in accordance with them, example first.
- You can figure out why code is the way it is by starting from a commit or even a single line, jumping to the session that produced it and recovering the full context of the decision.
- You can resume someone else's in-progress work on your own machine and carry it forward.
- You can point an agent at a specific past session to fix a bug or to continue a half-built feature.
- You can answer questions about people, not just code: who is the domain expert for the kind of change you are about to make, and who should review it.
Nearly none of these are possible with semantic memory. A pile of facts cannot tell you what was rejected and why. It cannot be spun up and interrogated. Only a narrow slice, some best-practices lookups, survives the translation into facts.
Why code is different from everything else
This is the main argument of this post. Semantic memory is genuinely useful in most domains. It is much weaker for code, and the reason is structural.
Coding produces hard artifacts. A Slack thread does not. When your team debates a decision in chat, there is no single correct answer sitting in a repo you can go check. In code there usually is, and it lives in the code. Getting a factoid that "GitHub timeouts should be handled with backoff" is strictly worse than getting the actual example of how timeouts are handled in your codebase, which an agent can read and replicate exactly.
Most coding is formulaic. When work follows established patterns, the ability to replicate an existing solution closely is what produces consistency and maintainability. A reference to the code that created the original is worth more than an abstract restatement of it.
And drift is real in code in a way it is not elsewhere. A document describing who owns what in your org is never truly correct. It is only ever an approximation of a fuzzy reality, and everyone reads it as such. But code has a correct answer at every moment, so an approximation of it is not just imperfect, it is actively misleading. A stale fact about your codebase does not degrade gracefully. It sends the agent down the wrong path with full confidence.
None of this means PRDs and design docs are worthless. Semantic memory has its place, including for code. The point is that software enjoys an enormous advantage no other domain has: the ground truth is right here, analyzable and executable. It can produce higher-quality, more current answers than any distilled artifact ever could. The memory that matters most for code is the memory closest to the code. Everything else is a map, and you should always be able to walk out and check the territory.
Where Polygraph sits
Now, let's see what Polygraph, an agent-agnostic meta-harness, supports. Today it does two things:
- It helps with working memory, the orchestration layer, with more planned.
- It handles the episodic-by-code cell: the recorded history of how code came to be.
Semantic-by-code is on the roadmap, but episodic comes first on purpose, and everything above explains why that ordering is not an accident.
Three graphs instead of one bag of facts
Most retrieval systems either use RAG, embedding everything and pulling back the nearest matches, or a knowledge graph of extracted facts. Both are built to serve semantic memory, a pile of statements you query.
Polygraph does something different because episodic memory has a different shape. It uses a combination of graphs plus RAG to retrieve and manage episodes. There are three graphs, and they lean on each other: a session graph, where each node is a unit of work; a repository graph, where the nodes are repos; and a code change graph.
The repository graph shapes the session graph. If two sessions touch the same repositories, they are far more likely to be related, so their proximity in the repo graph strengthens the edge between them in the session graph.
Other forces pull the session graph into shape too. Similarity of descriptions, logs, and commit messages. The developers who participated. Similarity of the actual code changes. And, of course, time, because a session from last week is usually more relevant than one from three years ago. As sessions accumulate, certain clusters of repositories keep showing up together, and the system draws new edges between them.
The payoff is retrieval that understands locality without being trapped by it. Most of the episodes you need will be close by, sitting near your current work in the graph. But some of the ones that matter most will be surprising and far away, a session in a repo you have never opened that solved exactly the problem you are solving now.
