Distributional Co-Founder & CEO Scott Clark recently led a lightning lesson hosted by Jason Liu as part of his series of talks helping builders successfully develop, deploy, and scale AI.
The observability hierarchy for AI systems has three distinct layers: logging and tracing to understand what happened in a specific session, monitoring and evals to determine whether the system is up and passing defined checks, and behavioral analytics to understand patterns across populations of agents and users. This talk focused on the analytics layer of AI observability. You can watch the lesson recording here:
And here are a set of frequently asked questions that Jason and Scott answered during this lesson.
What is the main difference between analytics and monitoring for AI systems?
Monitoring tells you if your system is up and whether your evals are passing at a high level. Logging and tracing help you debug specific sessions. Analytics fills the gap between these extremes by helping you discover, understand, track, and prioritize hidden behavioral signals across many sessions. Instead of looking at aggregate statistics or individual traces, analytics examines behavioral patterns across populations of agents to surface issues you didn’t know to look for.
What is Distributional?
Distributional is an AI analytics platform. It is a free, open tool that deploys on premises to analyze production AI logs. The platform uses unsupervised learning to discover behavioral patterns in agentic systems, surfacing insights with supporting evidence and suggested directions for fixes. It is designed for product owners and engineers who want to understand what is actually happening with usage and behavior of AI agents at scale.
Why did Distributional choose an analytics-focused approach instead of just testing?
It is impossible to anticipate everything that can happen in production ahead of time. Users will interact with systems in unexpected ways, and foundational models are non-stationary and continuously changing. You can only observe emergent behaviors by looking at real production data. Analytics helps discover unknown unknowns, which can then be turned into known issues that are tested for going forward.
How does Distributional’s behavioral analytics work technically?
Distributional enriches trace data with behavioral signals such as LLM-as-judge evals and statistical measures. Each trace is represented as a behavioral vector in high-dimensional space describing what actually happened. The analysis phase examines distributions of these vectors to identify subclusters representing infrequent behaviors or patterns correlated with cost, latency, or quality issues. These subclusters are then analyzed using LLMs to generate insights and suggest areas for improvement.
What are the benefits of this approach for developers?
Distributional provides a guided or “tab-complete” analytics experience, where insights are presented without requiring weeks of manual data science work. Developers can quickly triage whether an issue matters, inspect specific evidence from traces, explore potential fixes ranging from prompt changes to architectural improvements, and track whether changes improve behavior over time. The system helps find needles in the haystack without requiring teams to know in advance what to look for.
Is Distributional’s analytics platform publicly available?
Yes. Distributional is free and openly distributed at docs.dbnl.com. It can be deployed on a Kubernetes cluster for full production use or as a lightweight K3D cluster packaged in a single Docker image for single-machine sandbox use. The sandbox can run locally and be operational in under an hour. All deployments run on-premise, meaning Distributional never sees your underlying data.
What types of insights does Distributional surface?
Distributional surfaces behavioral patterns that emerge in production, such as redundant tool usage within a single session, failures that appear when systems are rolled out to new regions with different terminology, non-linear failures caused by adding more tools or MCP servers, rare edge cases like intermittent 403 errors, and agents getting stuck in tool-calling loops. Each signal is coupled with concrete evidence and contextual guidance on how teams might address the issue or improvement.
How do you integrate data into Distributional?
Many agent frameworks already emit OpenTelemetry traces, which can be routed to Distributional in the same way they would be sent to tools like Datadog or CloudWatch using a write-once, send-many approach. Teams with existing ETL pipelines can also ingest data via Parquet files, Iceberg tables, or SQL. Richer data – such as tracing details, session-level events, user feedback, and eval outputs – enables deeper and more useful analysis.
How does Distributional handle the relationship between evals and production analytics?
Distributional treats analytics and evals as complementary parts of a continuous improvement flywheel. Production analytics is used to observe real-world behavior and discover previously unknown failure modes. Those discoveries can then be converted into new evals or reward signals, improving system behavior over time. Passing evals does not guarantee real-world performance, and analytics helps reveal what existing evals are missing.
What does Distributional believe about the future of AI observability?
Analytics is a natural and inevitable stage in the AI development lifecycle, following the same pattern seen in previous software paradigms. Teams build systems first, then add logging, then monitoring, and eventually analytics to extract maximum value and understanding. The question is not whether behavioral analytics will be needed for agents, but when teams will invest in it to complement their monitoring and debugging tools.
Get started
Distributional is a free, open, and installable platform for agent analytics. Try it today and quickly learn how it complements your existing agent observability stack. We are also always happy to learn more about your use case and enterprise needs, so reach out to contact@distributional.com with any questions.
Subscribe to DBNL
Thank you for your submission!
Oops! Something went wrong while submitting the form.
