Agent Memory Expert

Overview

A skill for designing, implementing, and optimizing memory systems for AI agents. Covers short-term context windows, long-term persistent memory, episodic recall, semantic search, and memory-augmented generation (MAG) — enabling agents that learn from past interactions and maintain coherent state across sessions.

When to Use

• Building agents that need to remember past conversations
• Implementing RAG (Retrieval-Augmented Generation) systems
• Designing multi-session context persistence
• Optimizing token usage through intelligent memory management
• Creating agents that improve from experience

Quick Start

Copy
# Design a memory system
claude "Design a memory architecture for a customer support agent that remembers past tickets"

# Implement persistent memory
claude "Add long-term memory to this agent using vector embeddings"

# Optimize context window usage
claude "Optimize this agent's memory to fit within 100K context window"

Memory Architecture

Memory Types

Type	Retention	Access Speed	Use Case
Working Memory	Current session	Instant	Active conversation context
Short-Term Memory	Hours	Fast	Recent interactions, session state
Long-Term Memory	Persistent	Medium	Knowledge base, past decisions
Episodic Memory	Persistent	Medium	Specific past interactions
Semantic Memory	Persistent	Medium	General knowledge, patterns
Procedural Memory	Persistent	Fast	Learned procedures, workflows

Architecture Diagram

┌─────────────────────────────────────┐
│         Agent Context Window         │
│  ┌──────────┐  ┌──────────────────┐ │
│  │ System   │  │ Working Memory   │ │
│  │ Prompt   │  │ (current task)   │ │
│  └──────────┘  └──────────────────┘ │
│  ┌──────────────────────────────────┐│
│  │ Retrieved Memory (relevant past) ││
│  └──────────────────────────────────┘│
│  ┌──────────────────────────────────┐│
│  │ User Message + Current Context   ││
│  └──────────────────────────────────┘│
└─────────────────────────────────────┘
         ↕ Retrieval / Storage
┌─────────────────────────────────────┐
│       Long-Term Memory Store         │
│  ┌──────────┐  ┌──────────────────┐ │
│  │ Vector   │  │ Key-Value Store  │ │
│  │ Database │  │ (facts, prefs)   │ │
│  └──────────┘  └──────────────────┘ │
│  ┌──────────┐  ┌──────────────────┐ │
│  │ Episode  │  │ Conversation     │ │
│  │ Index    │  │ Summaries        │ │
│  └──────────┘  └──────────────────┘ │
└─────────────────────────────────────┘

Implementation Patterns

1. Context Window Management

Copy
class ContextManager {
  maxTokens: number = 100000;
  reservedForResponse: number = 4000;
  systemPromptTokens: number = 2000;

  buildContext(memories: Memory[], currentMessage: string): string[] {
    let budget = this.maxTokens - this.reservedForResponse - this.systemPromptTokens;

    const sections = [];

    // Priority 1: Current conversation (most recent messages)
    const recentMessages = this.getRecentMessages(5);
    budget -= this.tokenCount(recentMessages);
    sections.push(recentMessages);

    // Priority 2: Retrieved relevant memories
    const relevant = this.rankByRelevance(memories, currentMessage);
    for (const memory of relevant) {
      const tokens = this.tokenCount(memory.content);
      if (tokens > budget) break;
      budget -= tokens;
      sections.push(memory.content);
    }

    // Priority 3: Summary of older context (if budget allows)
    if (budget > 500) {
      sections.push(this.generateSummary(budget));
    }

    return sections;
  }
}

2. Vector-Based Retrieval

Copy
import { embed, cosineSimilarity } from './embeddings';

class VectorMemory {
  async store(content: string, metadata: Record<string, any>) {
    const embedding = await embed(content);
    await this.db.insert({
      content,
      embedding,
      metadata,
      timestamp: new Date(),
    });
  }

  async retrieve(query: string, topK = 5): Promise<Memory[]> {
    const queryEmbedding = await embed(query);
    const results = await this.db.similaritySearch(queryEmbedding, topK);
    return results.filter(r => r.score > 0.7); // Relevance threshold
  }
}

3. Conversation Summarization

Copy
async function summarizeConversation(messages: Message[]): Promise<string> {
  // Summarize when conversation exceeds threshold
  if (messages.length < 10) return '';

  const oldMessages = messages.slice(0, -5); // Keep last 5 intact
  const summary = await llm.summarize(oldMessages, {
    instruction: 'Summarize key decisions, facts learned, and action items',
    maxTokens: 500,
  });

  return `## Previous Conversation Summary\n${summary}`;
}

4. Episodic Memory

Copy
interface Episode {
  id: string;
  timestamp: Date;
  trigger: string;        // What initiated the episode
  context: string;        // Relevant context at the time
  actions: string[];      // What was done
  outcome: string;        // Result
  lessons: string[];      // What was learned
}

class EpisodicMemory {
  async recordEpisode(episode: Episode) {
    await this.store(episode);
    // Also create searchable embedding
    const summary = `${episode.trigger}: ${episode.outcome}`;
    await this.vectorStore.store(summary, { episodeId: episode.id });
  }

  async recallSimilar(situation: string): Promise<Episode[]> {
    return this.vectorStore.retrieve(situation, 3);
  }
}

Memory Storage Options

Storage	Best For	Pros	Cons
CLAUDE.md	Project facts, preferences	Auto-loaded, simple	Limited size
SQLite	Structured data, episodes	Fast local queries	Not distributed
PostgreSQL + pgvector	Production vector search	Full SQL + vectors	Setup complexity
Pinecone	Large-scale vector search	Managed, scalable	External service
ChromaDB	Local development	Easy setup, in-memory	Not for production
Redis	Short-term cache	Ultra-fast	Volatile by default
File system	Simple persistence	Zero dependencies	No search capabilities

Memory Lifecycle

Storage Strategy

Event occurs → Evaluate importance → Store if significant
                                  → Discard if trivial

Importance factors:
- User explicitly asked to remember
- Decision was made with rationale
- Error was encountered and solved
- Pattern was confirmed across interactions
- Preference was expressed

Retrieval Strategy

New query → Extract key concepts → Search memory
         → Rank by relevance + recency + importance
         → Select top results within token budget
         → Inject into context

Forgetting Strategy

Periodic review → Score memories by:
  - Recency: When was it last accessed?
  - Frequency: How often is it retrieved?
  - Relevance: Does it match current project?
  - Accuracy: Is it still correct?
→ Archive low-scoring memories
→ Delete obsolete or incorrect ones

Best Practices

1. Summarize, don't store verbatim — Compress conversations into key facts and decisions
2. Index with metadata — Add timestamps, topics, and importance scores for better retrieval
3. Set relevance thresholds — Don't inject memories below a similarity score of 0.7
4. Manage token budgets — Always reserve space for the response and current context
5. Update over append — Modify existing memories when information changes
6. Separate facts from opinions — Tag memories as factual vs preferential
7. Version your memories — Track when memories were created and last updated
8. Test retrieval quality — Regularly verify that relevant memories are actually being found
9. Handle conflicts — When memories contradict, prefer the most recent
10. Privacy-aware — Don't store sensitive data without explicit consent