Workflow: Understand AI product behavior

Distributional automates analysis of enriched production AI logs to surface interesting signals of AI product behavior—the interrelation of inputs, prompts, context, tools, model, and response. This product is composed of three components: platform, pipeline, and workflow. The platform enables you to deploy, administer, scale, and integrate our product seamlessly with your stack. The pipeline powers this workflow with automation of data analysis at scale. And the workflow empowers you to leverage insights to continuously improve your AI products. 

Our goal with this workflow is to empower you to understand AI product behavior, and for this to be a launching point for continuous improvement of your AI products. This workflow empowers your AI product teams with three daily steps. 

Discover signals in context of standard usage patterns

The starting point for your interaction with Distributional is to discover new signals on AI product behavior. With the first two weeks of data, Distributional learns a standard set of topics and canonical pathways a user takes through your AI product—including all of the data sources, LLM requests, and tool calls that define intermediate steps in this flow. Going forward, we classify future daily usage according to this rubric, so you always know how yesterday’s behavior compares to this standard. 

Alongside this daily assessment of AI product status, Distributional is uncovering behavioral signals hidden in your AI production logs. These signals include increased propensity for a given topic, anomalous response quality as assessed by LLM as judge, changes in the distributions of metrics that define AI product performance, or interesting correlations between clusters of multiple properties. Distributional provides these signals—as well as any alerts on previously defined threshold violations—to the user in a daily report that serves as the starting point for deeper understanding of AI product behavior.

Investigate signals with curated evidence

Distributional links each signal to a curated set of evidence to the claim. Although the signals themselves are distilled sentences explaining the discovery, the evidence includes detailed historical analysis and a specific subset of logs from which the signal was derived. Distributional also places these signals in the context of topics and flows that define canonical usage patterns. 

This approach is particularly important as AI product usage scales. Without unsupervised analysis that points you in the right direction and surfaces the relevant traces for deeper analysis, your team is stuck randomly sampling or reviewing only human feedback. The goal is to take this evidence and take action to improve or debug your product, so these signals drive product improvement over time.

Track signals to validate product behavior

As you investigate these signals and take action to improve your product, Distributional makes it easy to identify or apply thresholds to these signals as new tracked metrics that define your AI product behavior. With a click of a button, you can add these metrics to Distributional, and they’ll appear as alerts via notification channels and as tracked metrics on the dashboard going forward. Each time you indicate your preference for tracking one of these signals, Distributional reinforces these preferences by tailoring future analysis. 

As you use Distributional, you develop a richer, more complete definition of AI product behavior that evolves with shifting usage patterns over time. And, most importantly, it aligns your team on their shared definition of performance. 

Try it today

This workflow is designed to give your AI product team a manageable starting point for improving your AI product every day. Distributional discovers new behavioral signals on your AI product daily. It provides curated evidence so you can quickly investigate and triage these signals. And it makes it easy to track your signals so you can validate any changes, improvements, or shifts in AI product usage. This workflow gives you a better understanding of your AI product as it evolves over time. 

Distributional’s full service is open and free to use. Try it today to experience these enterprise features yourself. We are also always happy to learn more about your use case and enterprise needs, so reach out to nick-dbnl@distributional.com with any questions.