What’s the real impact of your AI agent?

The analytics layer between LLM observability and product analytics. Every user message, tool call, agent response, and session end becomes an Amplitude event tagged with topic, quality score, and behavioral pattern, so you can build cohorts, funnels, and retention curves on AI session quality.

Most observability tools today let you answer questions such as

• "How performant is my product (latency in seconds)?"
• "Do we see infrastructure failures or technical errors?"

They're not designed to help answer questions such as

• "How did users navigate my digital experience before, during and after using my Agent?"
• "What is the conversion rate for OpenAI vs Anthropic vs Google AI models in production?"
• "What failure modes impact retention?"

Even the observability tools that do show traces, don't make it easy to filter them for behavioral cohorts (eg. "Show me traces of power users or people that dropped off at step 3 of a funnel").

Amplitude Agent Analytics helps answer both is designed specifically for AI builders, product managers and ML Ops Engineers to score online production sessions on AI quality by defining evalulations (evals) and also crucially connect AI quality to product outcomes including conversion, retention and revenue.

Yes. We know how important it is to measure AI products so every Amplitude plan, including our starter (free), gets a default allocation to use Agent Analytics.

Your monthly agent session allowance scales by tier: 5,000 on Free and Plus, 10,000 on Growth, and 20,000 on Enterprise. Take advantage of the full evaluation stack that ships at every tier, including out-of-the-box signals, trace analysis, semantic filtering, topic clustering. Define custom LLM evaluators (evals), calibration runs, and Bring Your Own Key (BYOK) for LLM calibration runs and custom evals. Growth and Enterprise extend agent session chat retention from 14 to 30 days, along with role-based access controls at Enterprise plan tier.

Amplitude doesn't charge for custom evaluations (evals). Bring Your Own Key (BYOK) for the model providers you already use to power your choice of LLM-as-a-Judge or Code-based custom evals.

Get started today.

Role-based access and privacy redaction are on by default.

If a user drops a phone number or another sensitive data that can be classified as PII into a support chat, Agent Analytics handles this at the SDK level, through a privacy mode you set. There are three modes:

• full is the default, and it sends prompt and response text with Personally Identifiable Information (PII) redaction on by default. This is completely configurable and extensible by customers. You can redact as needed beyond the out of the box PII scrubbing rules that the SDK ships with.
• metadata_only sends tokens, latency, model, and cost, and no text at all, which is the setting for regulated or sensitive workloads.
• customer_enriched sends pre-scored summaries from your own evaluation stack and nothing else.

Some examples of fields we redact by default are Email, Phone, Credit Card, SSN, IP Address. When these values are sent into Amplitude Agent Analytics, the resulting traces will show those as [email], [phone], [credit_card], [ssn], [ip_address] so the actual values are obfuscated.

Additionally, Access to Agent Analytics runs through Amplitude’s role-based access control. Three permission levels are available:

• View Agent Analytics Objects lets a role open sessions, evaluators, and related objects. A role needs this to use the product at all.
• Manage Inactive Evals and Runs lets a role create evaluators and launch dry runs.
• Activate Evals lets a role turn an evaluator live or archive one.

You decide which roles carry which permission, so a support analyst can read sessions while only your ML leads can push an evaluator into production.

Take advantage of enterprise-grade controls when sending traces to Amplitude Agent Analytics. Read more here.

Python on PyPI (pip install amplitude-ai) and Node.js / TypeScript on npm (npm install @amplitude/ai). Native wrappers for OpenAI, Anthropic, Gemini, Bedrock, Mistral, and Azure OpenAI. Framework integrations for LangChain, LlamaIndex, OpenAI Agents SDK, CrewAI, and the Claude Agent SDK. Anything emitting OpenTelemetry GenAI spans (OpenLIT, Traceloop, and OpenAI instrumentation) flows in via the bridge. Learn more here.

Every agent event carries a Session Replay ID. From any session in the explorer, View Replay opens at the moment the conversation started, so you watch the agent fail inside the actual product the user was using.

All agent interactions are available as standard behaviors to query alongside classic events in Amplitude charts. They're also available to use through the rest of Amplitude's platform including Experiment and Guides. Each event carries the experiment variant as a property, so prompt A/B tests attribute correctly across multi-turn conversations. Quality scores and behavioral patterns flow into cohorts that Guides and Activation target in real time.

The answer depends on what you're using in those other tools.

• If you're using them only to score online production sessions and nothing else, Agent Analytics will help serve your needs and you can replace those other tools. You'll be able to take advantage of out-of-the-box Signals, standard evaluations (evals) of task completion, response quality, user intent, session safety, user friction, negative feedback, and data quality. Explore topics across all agent conversations to quickly learn AI quality measures and what parts of your agent needs immediate fixing. You'll also crucially then connect AI quality to product outcomes including conversion, retention and revenue.
• If you're using aspects like offline evals or deploying agents using those tools, then keep them. Agent Analytics is not designed for those specific use cases.

Try Agent Analytics today as it is available on all plans, including our starter (free), to all customers. Send us a few traces, filter them by behavorial cohorts, build a funnel or retention chart with agent behaviors. Get started today.

Yes. Ranging from Y Combinator startups such as Hyrbd and Cashbook to enterprise companies such as The Economist, Agent Analytics has been put through rigorous performance tests. We originally designed Agent Analtyics for ourselves and have been documenting our own usage of it here and here.