Memory Is Not Storage

While working on agent memory, I saw a remembered fact remain available after the world it described had changed. Retrieval returned exactly what had been captured. The problem was that the fact was no longer current. Recall worked; current truth did not.

The governing question is: how can past experience help without quietly becoming authority?

Note: This is a design position shaped by building Waldo and reading current work on long-term agent memory. It has not yet been validated as a product result.

Consider telling an agent, “I prefer calls after lunch.”

That could be a permanent preference. It could be true for one week. It could apply only to work calls. It could even be a polite excuse you do not want turned into a rule.

Saving the sentence is easy.

Knowing when it should affect a future action is the memory problem.

Retrieval is only one step

Most descriptions of agent memory sound like this:

capture → store → retrieve

A personal agent needs a longer loop:

observe → propose → confirm → use → inspect
             ↑         ↓        ↓
          correct    expire   revoke

Each step raises a different question.

• Did the user say this, or did the agent infer it?
• Is it a fact, a preference, or an interpretation?
• How long should it remain current?
• Is it allowed to influence this decision?
• Can the person see and change it later?

A vector database can help retrieve information. It cannot answer these questions on its own.

Recent work on governed evolving memory treats memory as changing state rather than passive storage. Broader surveys of agent memory systems and memory for language agents make a similar shift: memory has to be understood as part of the agent’s behavior.

That framing makes sense to me. A remembered preference can change a schedule. A remembered relationship can change the tone of a message. A remembered commitment can become an open loop.

Once memory changes action, it needs rules.

Four distinctions matter

What I said and what the agent inferred

“I do not want to work late this week” is a direct statement.

“Shivansh is most productive in the morning” may be an inference from behavior.

They should not look the same. What a person says about themselves should outrank the profile an agent builds around them.

An event and its interpretation

“The meeting moved to Friday” is an event.

“This relationship is deteriorating” is an interpretation.

The interpretation may be useful, but the person should be able to see what it came from and disagree with it.

Recall and authority

An agent may remember that I usually choose an aisle seat. It can use that when comparing options.

That does not give it permission to buy a ticket.

Memory can inform an action without authorizing it.

History and current truth

A memory can be accurate and still be out of date.

Projects end. Goals change. Relationships shift. Temporary constraints expire. The system needs a way to represent time and supersession, not only similarity to the current prompt.

What a useful memory should carry

A memory does not need to expose a database schema to the user. But it should preserve enough context to explain itself.

memory:       Prefer calls after lunch
source:       User said it directly
scope:        Work calls
captured:     13 August 2026
status:       Current until corrected
authority:    A scheduling preference, not permission to book

The exact fields will change. The important part is that the system can answer: why do I remember this, and how did it affect what I did?

Contradiction is normal

People are inconsistent because context changes.

Someone can protect deep-work mornings and still take an urgent morning call. They can care about health without wanting health data connected. They can prefer short work updates and long research conversations.

The agent should not silently average these into one personality profile.

When memories disagree, it should consider which source is newer, which is more direct, whether the context changed, and whether the consequence is important enough to ask.

For a small, reversible choice, it can make a sensible default and say what it assumed. For a consequential choice, ambiguity should slow it down.

Correction has to change behavior

An editable memory list is not enough.

Suppose the agent thinks an old startup is still active. I correct it. The correction should also update plans, recommendations, and open loops that depended on the old belief.

That gives us a useful test:

After a correction, can the user find every important place where the old belief still has influence?

If not, the system remembers the correction but does not actually learn from it.

Forgetting is part of the product

Total recall sounds useful until it is applied to a life.

Some details should expire. Some should be compressed into a conclusion. Some should remain as history but stop influencing current behavior. Some should disappear because the user withdrew consent.

These are different actions:

• Expire: this was temporary and is no longer current.
• Supersede: keep the history, but prefer the newer statement.
• Compress: keep the conclusion without carrying every raw detail forward.
• Revoke: stop letting this memory influence behavior.
• Delete: remove the retained data where the system genuinely can.

Forgetting is not a failure of memory. It is part of agency.

Memory is also a security boundary

Long-lived memory lets mistakes travel across time.

Incorrect or malicious information can outlive the session that introduced it and influence unrelated actions weeks later. A review of long-term memory security describes risks across that lifecycle, including poisoning and unsafe retention.

For a personal agent, the sources are especially mixed: messages, files, tool output, calendars, health data, and the agent’s own inference.

“The model saw it” cannot mean “the user approved it as memory.”

What this means for Waldo

I want Waldo’s memory to reduce reconstruction without becoming surveillance.

That leads to a few simple rules:

• what the user says outranks what the agent infers;
• sensitive context is permissioned, never assumed;
• memory and permission remain separate;
• important recommendations can be traced to current evidence;
• corrections reach future behavior;
• the user can release things they no longer want to carry.

The approach may still be too heavy. People may not want to manage provenance and scope every day. The right interface probably keeps most of this quiet and brings it forward only when the consequence matters.

The real test is not whether the agent can recall more. It is whether the person repeats themselves less, corrects the agent successfully, and understands why it acted.

The question is not: how can an agent store more of me?

It is: how can past experience help without quietly becoming authority?