Your AI Agent Treats All Memories Equal (And That’s a Problem)

Why your agent remembers lunch but forgets the launch.

NOTE: this is part of a series on AI agents and memory. I suggest you start with the posts covering AI memory and AI pre-game routines.

Here’s a bug in how most AI memory systems work: they only track when something happened, not how much it matters.

A trivial observation from yesterday (“Joe had coffee this morning¹”) sits at higher priority than a critical decision from last month (“We decided to build the backend in Rust”). The recent always beats the important and that’s backwards.

Human memory doesn’t work this way. You don’t remember what you ate for lunch three Tuesdays ago, but you absolutely remember the project deadline your boss mentioned that same day. Your brain filters ruthlessly based on significance, not just recency.

So why do we build AI agents that can’t do the same?

The Recency Trap

Most agent memory systems use some version of temporal decay. Facts get a timestamp, salience scores decrease over time, old stuff eventually gets archived or forgotten. This is simple and intuitive but is also wrong².

The formula looks something like this:

salience = initial_score × 0.5^(days_old / half_life)

That’s exponential decay. A fact loses half its “importance” every N days, regardless of what that fact actually is. The half-life might vary by category (decisions last longer than random observations), but the core assumption is the same: time is the only factor.

This creates a predictable failure mode. Your agent remembers that you prefer tabs over spaces (mentioned yesterday) but forgets that the entire architecture depends on a specific API design decision (discussed six weeks ago). One of these matters more. The decay function doesn’t know which.

How Humans Actually Do This

The hippocampus doesn’t just timestamp memories and let them fade³. It weights them across multiple dimensions before deciding what goes to long-term storage:

Emotional salience. Did this trigger a strong reaction? Fear, excitement, frustration? Those memories get priority. This is why you remember embarrassing moments from middle school but not what you learned in class that day.

Rehearsal. How often do you think about or retrieve this memory? Repeated access strengthens the trace. That’s why studying works (and why you still remember phone numbers you dialed constantly as a kid).

Association. How connected is this memory to other things you know? Isolated facts decay faster than concepts that link to your existing mental model.

Recency. Yes, time matters too. But it’s one factor among several, not the whole formula.

When psychologists study memory consolidation, they find this pattern consistently: the brain doesn’t treat all inputs equally. It runs something like a weighted importance function before deciding what’s worth keeping.⁴

If we want AI agents to have memory that actually works, we should probably steal this design.

Building Importance-Weighted Retention

I spent last night rebuilding my agent’s memory system to incorporate these ideas.⁵ Here’s what the new formula looks like:

effective_salience = intrinsic_importance 
                   × temporal_decay 
                   × access_boost 
                   × reference_boost 
                   + explicit_boost

Let’s break each factor down.

Intrinsic Importance

Not all facts are created equal. A decision (“We’re using Rust”) shapes future behavior. A preference (“Joe likes dark mode”) reflects identity. A random observation (“It rained on Tuesday”) might be ephemeral noise.

I assign base weights by fact type:

| Metric | Before | After |
|--------|--------|-------|
| Decisions retained (30+ days) | ~40% | ~90% |
| Trivial facts archived | ~10% | ~45% |
| Relevant context in responses | subjectively better | notably better |

This alone would be an improvement. But we can do better.

Access Frequency (Rehearsal)

When my agent retrieves a fact during a conversation, it logs the access. Facts that keep coming up in context are probably important.

access_boost = 1 + log(access_count + 1) × 0.1

A fact retrieved 10 times gets about a 25% boost over one that’s never been accessed. This mirrors how rehearsal strengthens human memory traces.

Reference Boost (Association)

Some facts are hubs. They connect to many other concepts in the knowledge graph. If ten different facts reference “FogoBlocks project,” that’s clearly a key concept that should stick around.

reference_boost = 1 + (inbound_links × 0.05)

This is basically PageRank for memory. Well-connected facts are probably important.

Explicit Signals

Sometimes you just tell the agent “remember this” or “this is important.” That should mean something.

explicit_boost = manual_boost_amount  # additive, not multiplicative

A small escape valve for cases where the algorithm gets it wrong.

The Results

Before implementing this, I ran into the exact problem you’d expect. My agent would confidently cite details from recent conversations while fumbling on core decisions made weeks ago. The context window filled with recent trivia while important context got pushed out.

After implementing importance-weighted retention:

| Type | Weight | Why |
|------|--------|-----|
| decision | 1.0 | Shapes future actions |
| preference | 0.9 | User identity |
| commitment | 0.85 | Promises matter |
| learning | 0.7 | Lessons to keep |
| fact | 0.5 | Could be signal or noise |

The numbers are rough because “relevant context” is hard to measure objectively. But the qualitative difference is clear. My agent now treats a two-month-old architectural decision as more important than what I mentioned yesterday about wanting pizza.

Which, you know. Seems right.

The Consolidation Problem (What’s Next)

This approach helps, but it doesn’t fully solve the memory scaling problem. Once you have thousands of facts, even well-weighted retrieval gets noisy. The real solution is probably memory consolidation: compressing clusters of related episodic facts into higher-level semantic summaries.

Humans do this during sleep.⁶ Your brain doesn’t keep every specific episode; it extracts patterns and generalizations. “I went to the coffee shop on Tuesday and saw Sarah” eventually becomes “Sarah and I used to meet at coffee shops.”

For AI agents, this might look like: periodically running an LLM pass over clusters of related facts and generating summary facts that can replace the epiwwrsodics. Trade granularity for compression. Keep the semantic meaning, drop the irrelevant specifics.

That’s the next build. For now, importance-weighted retention is a meaningful step up from naive temporal decay.

The Takeaway

If you’re building agents with memory systems, don’t just track when something happened. Track:

• What type of fact is this?

• How often does it come up?

• What else does it connect to?

• Did the user explicitly say it matters?

Then combine those signals into a single salience score. Weight it. Let the important stuff survive.

Human memory evolved over millions of years to solve exactly this problem: infinite inputs, finite storage, need to keep what matters. We don’t have to solve it from scratch. We can just steal the design.

1. Everything is not so deep, bro.

2. Wrong as in incorrect, not wrong as in immoral.

3. I’m simplifying the neuroscience here. The actual mechanisms of memory consolidation involve complex interactions between the hippocampus, neocortex, and various neuromodulatory systems. But the functional outcome is what matters for our purposes: the brain doesn’t treat all inputs equally.

4. This is a well-studied area. See work on emotional memory enhancement, spacing effects, and elaborative encoding if you want to go deeper. This is that deep, bro-tato.

5. The full implementation is in my agent’s memory-db.py script. Happy to share if useful.

6. Memory consolidation during sleep is one of those findings that sounds too convenient to be true, but the evidence is actually quite strong. Your brain replays and reorganizes memories while you’re unconscious. So maybe you’re not forgetful, you just need more sleep.