AI Agent Production: From Prototype to Reliable Service

Harness: Infrastructure, Not Just a Prompt

An agent is not just an LLM and a prompt. It is software infrastructure (harness) that orchestrates the agent's work: manages context, calls tools, stores memory, ensures control. The quality of the agent is determined by the quality of the harness.

The harness includes: code execution environment, tool registry, security services, monitoring and tracing, memory management. In a corporate environment, this is often separated into a dedicated platform (AI Hub).

Do not start by building your own harness. Use existing frameworks (LangGraph, LangChain) and adapt them to your requirements. Custom development is justified only for specific requirements.

• Harness: infrastructure for orchestrating the agent.
• Includes: runtime, tools, memory, security, monitoring.
• In a corporate environment — AI Hub.
• Use existing frameworks, do not build your own.

Runtime, Queue, Sandbox, and Credentials

The runtime is the environment in which the agent's code executes. This can be a container, a serverless function, or a virtual machine. The runtime must be isolated so the agent cannot harm other systems.

A message queue allows processing requests asynchronously. This is important for long-running tasks. The agent receives a task, puts it in the queue, and a handler executes it in the background. The client can poll the status.

A sandbox is an isolated environment for executing generated code. If the agent can generate code, it must run in a sandbox with limited permissions. Credentials must be stored in a secure store and not appear in the prompt.

• Runtime: isolated execution environment.
• Queue: asynchronous processing of long-running tasks.
• Sandbox: isolation of generated code.
• Credentials: secure store, not in the prompt.

• Define the execution environment: container, serverless, VM.
• Set up runtime isolation.
• Add a queue for long-running tasks.
• Set up a sandbox for generated code.
• Store credentials in a secure store.

External Effects and Idempotency

The checkpointer saves the graph state but does not guarantee that an external effect (sending an email, writing to a database) will be executed exactly once. If a node is re-executed (for example, after a failure and recovery), the side effect may repeat.

Therefore, external operations must be idempotent. Use unique operation identifiers so that a repeated call does not create a duplicate. For example, when creating a ticket, pass request_id, and the system will ignore repeated requests with the same ID.

Design tools with the possibility of repeated calls in mind. This is critical for reliability.

• The checkpointer does not guarantee exactly-once for external effects.
• A node may be re-executed after a failure.
• External operations must be idempotent.
• Use unique operation IDs.

FastAPI: Synchronous Call or Long-Running Task

A simple way to run an agent is to wrap it in a FastAPI endpoint. The client sends a request, the server runs the graph and returns the result. This is suitable for fast tasks that complete in seconds.

Long requests can hit client, proxy or server timeouts. A queued job with a task_id, persisted state and status lookup is often more reliable than a long-held HTTP request. This is not the same as declaring a handler async: asynchronous syntax does not provide job persistence or recovery.

The choice between synchronous and asynchronous calls depends on the task execution time and user experience requirements.

• FastAPI: synchronous call for fast tasks.
• Long tasks: queue + task_id + status polling.
• Synchronous calls are not suitable for minute-long tasks.
• The asynchronous pattern improves user experience.

1. Client -> POST /run (request)
2. Server: run the graph
3. Server -> Client: result (synchronously)
4. Client -> POST /run (request)
5. Server: put in queue, return task_id
6. Client -> GET /status/{task_id}
7. Server -> Client: status and result

Streaming vs. Persisted Result

Streaming allows returning intermediate results to the client: visited nodes, parts of the response. This improves the user experience, especially for long-running tasks. LangGraph supports streaming via the stream method.

However, streaming does not replace saving the result. If the client disconnects during streaming, the result should be available later. Save the final result to storage and allow the client to retrieve it by ID.

Streaming is a delivery method, not storage. Use both mechanisms: streaming for interactivity, persistence for reliability.

• Streaming: intermediate results in real-time.
• Streaming improves the user experience.
• Save the final result to storage.
• Streaming is delivery; persistence is reliability.

Concurrency, Backpressure, and Cancellation

When running in production, you need to consider concurrency: how many requests the agent can process simultaneously. This depends on LLM API limits, runtime capacity, and execution time.

Backpressure is a mechanism that protects the system from overload. If the request queue is too long, new requests are rejected or the client is informed to wait. This prevents denial of service.

Cancellation is the ability to stop task execution. If the client canceled the request or the task exceeded the time limit, the agent should stop. Implement a cancellation mechanism by checking a flag in graph nodes.

• Concurrency: how many requests to process simultaneously.
• Backpressure: protection from overload.
• Cancellation: stopping task execution.
• Check the cancellation flag in graph nodes.

Observability with Redaction

Observability is the ability to understand what the agent is doing. Log tool calls, node transitions, token usage. This allows debugging and analyzing behavior.

Logs may contain sensitive data: personal information, keys, email texts. Therefore, apply redaction before writing to the log: replace sensitive values with masks.

Observability should be built into the harness, not added post-hoc. Use structured logging and tracing.

• Log tool calls and transitions.
• Log token usage.
• Redact sensitive data in logs.
• Observability should be built into the harness.

• Log every tool call with parameters and result.
• Log transitions between graph nodes.
• Mask personal data and keys in logs.
• Set up tracing for end-to-end analysis.

Metrics and Evaluations: Precision, Recall, pass@k

Use metrics to evaluate agent quality. Precision is the proportion of correct positive predictions among all predicted positives. Recall is the proportion of correct positives among all true positives. F1 is the harmonic mean.

pass@k is the probability that at least one of k agent runs gives the correct result. This is useful for tasks where a single successful solution is enough (e.g., code generation).

Metrics have limitations. Precision and recall do not account for the cost of errors. pass@k does not account for the quality of partial solutions. Interpret metrics in the context of the task.

• Precision: accuracy of positive predictions.
• Recall: completeness of positive predictions.
• F1: harmonic mean of precision and recall.
• pass@k: probability of success of at least one of k runs.
• Interpret metrics in the context of the task.

Latency, Cost, and Tokens

Each LLM call has a cost and latency. Optimizing prompts (reducing tokens) lowers cost and speeds up responses. Use few-shot examples sparingly, remove irrelevant context.

Track token usage: tokens per request, per session, per user. This allows predicting costs and detecting anomalies.

Balance quality and cost. A more powerful model may give a better answer but cost more. Use routing: simple requests to a fast model, complex ones to a powerful model.

• Each LLM call has a cost and latency.
• Prompt optimization reduces cost.
• Track token usage.
• Routing: a simple model for simple tasks.

Acceptance Criterion: Task Result, Not Call Count

The success of an agent is determined not by the number of tool calls but by the quality of the result. The task is solved — the agent is successful. The task is not solved, even if the agent made 100 calls — it is a failure.

Define acceptance criteria for each task. For example, for the task 'answer a question about order status', the criterion is the correct status and date. For the task 'generate a report', the presence of all sections and data correctness.

Use acceptance criteria to evaluate and optimize the agent. If the agent often fails to achieve the goal, analyze at which stage the failure occurs.

• Success is the task result, not the number of calls.
• Define acceptance criteria for each task.
• Analyze at which stage the failure occurs.
• Optimize the agent against acceptance criteria.

Bounded Pilot and Failure Walkthrough

Launch the agent as a bounded pilot: a small group of users, a limited set of tasks. This allows identifying problems without large-scale consequences.

Before the pilot, define success metrics, feedback collection mechanisms, and a rollback plan. If the agent shows poor results, you should be able to quickly disable it.

A failure walkthrough is an incident analysis: what happened, why, how to prevent it. Use logs and tracing to reconstruct the sequence of events.

• Pilot: a small group, a limited set of tasks.
• Define success metrics and a rollback plan.
• Collect feedback from users.
• Failure walkthrough: incident analysis and prevention.