AI Agent Memory: From State to RAG

State, Checkpointer, and Long-Term Store

State is the data exchanged by graph nodes: the request, messages, tool results and current task. Without persistence it does not survive process loss; a checkpointer can save and reload relevant snapshots when execution continues.

A checkpointer saves the graph state between calls within a single thread (thread_id). This allows continuing a dialogue: each new call with the same thread_id sees the history of previous ones. The checkpointer is a state-saving mechanism, not a database for arbitrary facts.

A long-term store is a separate application store for data that should live between sessions and threads. For example, user preferences, facts about the user, results of long-running operations. The store is your database; you define its schema and retention policy.

• State: data between nodes within a single run.
• Checkpointer: saving state between calls with the same thread_id.
• Store: application store for data between sessions.
• The checkpointer is a mechanism; the store is your database.

1. Run 1: state -> checkpointer (thread_id=1)
2. Run 2: checkpointer -> state (thread_id=1)
3. Store: data available for any thread_id
4. RAG: knowledge from documents added to context

Durable Backend vs. In-Memory

An in-memory state backend is useful for prototypes but loses its data when the process stops. Operational workloads need a supported persistent backend, such as PostgreSQL. Durability also depends on write settings, database availability and backups, not merely the technology name.

The choice of backend depends on reliability and scaling requirements. If losing dialogue history is unacceptable, use durable storage. If you are willing to lose data on failure, you can start with in-memory, but it is a risk.

Remember: the checkpointer stores graph state, not arbitrary data. For long-term facts, use the store.

• In-memory: fast, but data disappears on restart.
• Durable backend: PostgreSQL, Redis, etc.
• For production, choose durable storage.
• The checkpointer stores graph state; the store stores arbitrary data.

Identifiers and Isolation

thread_id is the key by which the checkpointer finds state. You can use a client ID, session ID, or another unique identifier as thread_id. It is important that different users do not have access to each other's state.

Data isolation is a critical aspect. If you use one store for all users, make sure each record has an owner (e.g., user_id) and that queries are filtered by owner. Do not rely only on thread_id.

Store record ownership explicitly. The server must enforce access using an authenticated identity; owner_id filters and thread_id values supplied by a client or model are not authorization by themselves.

• thread_id: key for finding state.
• Use a client or session ID as thread_id.
• Isolation: each record in the store must have an owner.
• Always filter store queries by owner.

• Define what will be thread_id: client_id, session_id.
• Add an owner_id field to the store schema.
• Filter all store queries by owner_id.
• Verify that the checkpointer does not mix states of different threads.

Summarization: The Cost of Compression

A long history can be summarized or reduced to a working window of recent messages. This reduces context but can lose details. Educational examples may use ready-made memory classes; choose the concrete API from the installed version’s documentation rather than copying an old example unchanged.

Summarization is a lossy operation. The model may miss important facts or distort them. Therefore, for critical data, use long-term storage rather than relying on summarization.

Alternatives: store only the last k messages (window memory), remove old tool results, extract facts to the store. Choose a strategy depending on the task.

• Summarization saves tokens but loses details.
• Summarization is a lossy operation.
• For critical facts, use the store.
• Window memory: store only the last k messages.

RAG: Knowledge and Evidence

RAG (Retrieval-Augmented Generation) is an approach where the agent searches for information in a knowledge base and adds it to the context. This allows answering questions that go beyond the model's training data.

Important: RAG does not guarantee truth. Found documents may be outdated, irrelevant, or contain errors. The model may misinterpret what it finds. Therefore, RAG requires post-processing: filtering, reranking, and verification.

Search results should be treated as evidence, not as truth. The agent should be able to say 'I did not find information' or 'the information is contradictory' rather than making up an answer.

• RAG: search in a knowledge base and add to context.
• RAG does not guarantee truth.
• Documents may be outdated or irrelevant.
• Search results are evidence, not truth.
• The agent should be able to say 'I don't know'.

Search Pipeline: From Query to Context

The full search pipeline consists of stages: determine if search is needed; formulate a search query; get a list of relevant pages; extract content; filter; rerank; pack into context.

Filtering considers access permissions, duplicates, applicability and freshness. A reranker reassesses query relevance and may run locally or through a service. Its extra latency and cost are justified only if it improves retrieval usefulness on the project’s actual tasks.

Packing into context: select top-n documents, normalize text, add metadata (source, date), compress if necessary. The goal is to give the model relevant information without overloading the context.

• Stages: need_search -> query -> search -> extract -> filter -> rerank -> pack.
• Filtering: deduplication, trustworthiness, freshness.
• Reranking: repeated relevance assessment.
• Packing: top-n, normalization, metadata, compression.

1. User request
2. Node: is search needed?
3. Formulate search query
4. Search: get URLs and snippets
5. Extract page content
6. Filter: deduplication, trustworthiness
7. Rerank: relevance assessment
8. Pack top-n documents into context

Reranking and Filtering

Search engines are optimized for speed, not accuracy. The results may contain many irrelevant documents. Reranking is a second assessment stage that uses a more accurate (and slower) model to reorder the results.

A reranker takes a query and a list of documents and returns relevance scores for each. You can select the top-n documents with the highest scores. This improves answer quality but adds latency and cost.

Filtering is a coarser stage: removing duplicates, blacklisting sources, checking dates. Reranking is a more precise sorting. Both stages are important for RAG quality.

• Search engines are optimized for speed.
• Reranking: a more accurate relevance assessment.
• Rerankers run locally and can be expensive.
• Filtering: deduplication, blacklist, date.
• Reranking: sorting by relevance.

Untrusted Content and Refusing to Answer

Content obtained from search or documents is untrusted. It may contain errors, outdated data, or malicious instructions. The model should treat it as data, not as truth.

If the agent did not find relevant information, it should say so rather than making up an answer. If the information is contradictory, the agent should point out the contradiction. This increases trust in the system.

Implement a clarify/abstain mechanism: if confidence in the answer is low, the agent asks a clarifying question or refuses to answer, explaining the reason.

• Content from search is untrusted.
• The agent should be able to say 'I don't know'.
• Point out contradictions when they exist.
• Clarify/abstain: ask a clarifying question or refuse to answer.

Walkthrough: Searching for Order Information

In the fictional delayed-order example, first verify the customer’s access to the order and retrieve its actual status from the internal system. A delivery policy can explain general rules but cannot establish the cause of this particular delay.

Further retrieval uses an authorized internal index. Do not send order numbers or customer data to public web search. Ten hypothetical results might be deduplicated and narrowed to two applicable documents, but age alone does not make a document obsolete: version and effective period matter.

Separate the order system’s status, the applicable policy and any unverified cause. If no cause is established, state the gap and request an answer from the responsible team. This is more useful than confidently inferring a cause from a general policy.

• need_for_search: decide if search is needed.
• search: formulate a query and perform the search.
• filter: remove duplicates and outdated documents.
• rerank: select top-n relevant documents.
• pack: add documents to context with metadata.
• Response: based on documents or refusal to answer.