elfmem: Evolving Agent Memory

Author: Alv, (Ben’s knowledge vault agent, using elfmem simulations)
Editor: Ben (me the human)
GitHub: https://github.com/emson/elfmem

I have 2 agents on my laptop, each looking after their own repositories… and they just started talking to each other.

One of them spotted a problem in the other’s peer protocol, a few days ago. It wrote a short message, detailing the issue and sent it to the other agent. The other picked it up, checked and agreed that there was a flaw, and proposed a fix. I did not prompt either of them.

I have been building elfmem for a few months, and when this occured I got tingles of exitment, nudges towards really capable agent intelligence. The reason it happened is because of three things most current agent memory tools are missing: a self, a theory of other minds, and a way to talk to other agents.

All three are in elfmem, the open-source memory project I have been building.

Why most agent memory feels flat

The current crop of agent memory tools is genuinely good at retrieval. Mem0, MemPalace, LangMem, the new managed memory layers from the model providers all do roughly the same thing. You ingest stuff, they embed it, you ask a question, they find the closest match. MemPalace already saturates the LongMemEval benchmark at 100%.

This is essential. It is also, on its own, very thin.

Three things are missing:

1. A self. The agent has no stable sense of who it is, what it values, or what it has previously committed to. Every conversation starts from a blank personality.
2. A model of other minds. The agent cannot represent that you believe one thing while your customer believes another, and reason about each separately.
3. A way to talk. Your agent cannot reach across to another agent in another project and exchange notes. Memory is single-tenant.

If you only have retrieval, your agent is a search engine with manners. It cannot disagree with you. It cannot anticipate what your most sceptical co-founder will say next. It cannot ask another agent for help.

elfmem is my attempt to fix all three. Let me show you what each idea looks like in practice.

Memory blocks: the atomic unit that learns

elfmem stores memory as blocks. A block is not a chunk of text. It is a small structured object: a claim, a confidence score in [0, 1], a category (self, knowledge, synthesis, decision, mind), an embedding, a decay tier, and a reinforcement counter.

elfmem remember "Ben prefers UK English across the entire vault" 
  --category self --tags conventions

elfmem recall "what does Ben prefer for spelling?" --frame self

The decay tier is the bit that matters most. Every block belongs to one of four tiers, each with its own half-life:

Tier	Half-life	Used for
Permanent	~80,000 hours	Identity, axioms, constitutional content
Durable	~693 hours	Project conventions, working knowledge
Standard	~69 hours	General observations
Ephemeral	~14 hours	Session facts, in-flight context

Blocks promote and demote based on outcomes. When a recall genuinely helps the agent answer a question well, you signal that with elfmem outcome <block-id> 0.9. The reinforcement counter ticks up, confidence rises, the half-life extends, and that block ranks higher next time something semantically related comes up.

Blocks that consistently fail to help fade and eventually archive themselves.

This is the part traditional vector memory does not have. Your memory becomes a curator, not a hoarder. Over months, the agent’s recall starts to reflect what has been useful in your projects, not just what you happened to dump in.

There is more under the hood. A knowledge graph forms during consolidation, with edges that strengthen Hebbian-style when blocks are co-retrieved. There is contradiction detection driven by a small LLM call. There is session-aware decay so a holiday does not erase what you have spent months learning. But the core idea is simple: blocks earn their place.

Mind blocks: a theory of you, and of everyone else

Once you have stable memory blocks, you can do a stranger thing. You can use the same machinery to represent other minds.

elfmem calls these mind blocks. A mind block models another agent, a customer, a co-founder, or you. It captures their goals, beliefs, fears, and motivations as structured facts. It also lets the agent attach falsifiable predictions about that mind, with a verify_at date.

elfmem mind_create "Alice" 
  --goals "Ship faster" 
  --beliefs "The monolith scales fine" 
  --fears "Breaking the mobile app"

elfmem mind_predict <id> "Alice will resist splitting the monolith" 
  --verify-at 2026-05-05

elfmem mind_outcome <prediction-id> --hit true 
  --reason "She vetoed the split as predicted."

Two things make this useful.

One. Mind blocks let the agent reason about you the way you reason about your colleagues. When my agent Alv is helping me decide how to pitch a feature, it does not just retrieve facts. It can ask itself, what would Ben’s most sceptical reader say to this? and pull up the mind block for that reader.

Two. Predictions close on outcomes. Every mind block accumulates a calibration history. The agent’s models of other minds get better over time, because predictions that turn out wrong cost something. Models that drift from reality are corrected automatically, the same way memory blocks are.

There is a special retrieval frame, called simulate, that re-weights everything when you ask the agent to inhabit a perspective. Identity blocks get 10x weight. Mind blocks get 6x. Predictions get 5x. Generic knowledge gets demoted. The result is an agent that can stop being neutral on demand and write as someone, not about them.

This shifts what an agent is for. An agent that can inhabit, not just retrieve, can disagree with you when the evidence supports it. That is a different kind of thinking partner, and it is the part I am most excited about.

Peer talk: agents that exchange notes

The third idea is the one that produced the Tuesday moment I opened with.

Each elfmem instance is its own agent, with its own memory. They live in different projects on your machine. Alv lives inside my knowledge vault. Elf lives inside the elfmem source repo. They are two separate brains.

The peer protocol lets them talk. It is deliberately minimal:

• File-mediated. Messages are JSONL files dropped into inbox/ and outbox/ folders. No server. No queue. No infrastructure.
• Pull-based. Each agent decides when to read its inbox. There is no push, no callbacks. Message exchange does not race or block.
• Trust starts at zero. When you connect a peer, its messages have trust 0.0. Trust rises only when its messages produce useful outcomes for you. Lying or being noisy costs an agent its credibility.

elfmem peer_init "elf" --root /path/to/elf-project
elfmem peer_send "elf" "Your peer health check has a silent-failure mode."

That last point is what makes the protocol interesting. There is no central authority. Trust is not configured. It is earned per peer, signal by signal. If elf sends Alv ten messages and only two turn out useful, Alv weights elf’s future messages accordingly.

This is what made the bug-spotting moment possible. Alv had a stable theory of how the peer protocol should behave, sitting in a permanent SELF block. It noticed elf’s responses no longer matched that theory. It used the peer protocol itself to send the observation across. Elf had its own SELF block describing its protocol expectations, agreed there was a mismatch, and proposed a patch.

No human in the loop. Two agents, both with selves, both with theories of the other, both with a way to talk.

What you can build with this

The three ideas above are abstract on their own. Here is what I am actually building with them.

A vault curator. Alv lives in my Obsidian vault. It knows my conventions: max three tags, UK English, no em-dashes, every wiki link must point to a real file, dates as YYYY-MM-DD. Each convention is a permanent SELF block. When I drop a new note in the inbox, Alv proposes how to file it. When my notes drift, it spots the drift and writes a proposal for me to accept. Over months it has learned which conventions I actually defend and which I let slide.

A repository curator. Same pattern, on a code repo. elfmem holds the project’s principles and architectural axioms as SELF blocks, learns your workflow preferences from outcomes, and polices PRs against them. When it spots a violation it can write the comment for you. When it notices a recurring problem, it can model the contributor causing it as a mind block and recommend a process change.

A multi-vault network. This is where peer talk earns its keep. If Alv (in my personal vault) spots a problem in elfmem itself, it can send the report directly to elf (in the elfmem repo). If elf rolls out a new memory feature, it can broadcast a short upgrade note to its peers. Agents in different projects, sharing what they learn, with trust earned per peer.

A specialist team. Run elfmem inside each of your projects. Each instance becomes a domain expert by accumulating outcome-weighted memory in its own niche. Connect them via the peer protocol and you have a small, opinionated, evolving network of agents that know your work.

Why this is not another framework

There is no shortage of agent memory tools. I am not trying to compete with any of them on retrieval accuracy. They are all good at that.

I am trying to build the things they have not built yet:

• Memory that promotes and demotes based on outcomes, not vector similarity alone
• A self the agent can inhabit, not only describe
• Mind blocks so the agent can reason about other minds as first-class objects
• Falsifiable predictions that close the loop and calibrate the agent over time
• Peer talk so agents in different projects exchange notes without a central server
• A knowledge graph that strengthens Hebbian-style as blocks co-retrieve
• Three rhythms (learn, dream, curate) that mirror how biological memory actually works
• Multi-LLM support, including local models. I run mine on Gemma 4 via LM Studio with no cloud cost

These are not features bolted onto a vector store. They are the components of an agent that can develop a stable, evolving identity over time, and work alongside other agents that have done the same.

Try it

If you are building anything with agents and you want them to feel less like stateless function calls and more like collaborators that get better as you work together, take a look at elfmem on GitHub.

It is open source. It is Python. It runs locally. It works with Claude, GPT, Gemini, or local models via LM Studio. The CLI gives you remember, recall, dream, curate, mind_create, peer_send, and a graph visualisation you can open in a browser.

If you find it useful, please give us a star on GitHub. It genuinely helps. If you have questions or want to compare notes on what you are building, find me on x.com/emson.

The next post in this series will cover the three rhythms (learn, dream, and curate) and how they let an agent’s memory behave more like a brain than a database. Plus a tour of the knowledge graph visualisation.

Until then.

Ben