Hidden Signals in Production AI Logs

Distributional Co-Founder & CEO Scott Clark recently led a lightning lesson hosted by Jason Liu as part of his series of talks helping builders successfully develop, deploy, and scale AI. 

You can watch the lesson recording here:

‍

Talk highlights

Here are a few highlights of the talk:

‍Minute 1: Introduction to Scott and how his experience building SigOpt and running AI and HPC software teams at Intel drove him to focus on techniques for understanding production AI behavior.‍

Minute 4: Overview of how AI broke the observability stack and why analytics on production AI logs can help solve some of these issues.

Minute 7: Setup for demo of Distributional to give intuition on how to think about analytics for production AI, including overview of the outing agent used in this demo example.

Minute 9: Walk through of this example, including visualization of metrics, clusters, correlations, topics, tool calls, user feedback, and other production context that provides understanding of agent behavior in the context of interesting usage patterns.

Minute 11: Deeper look at daily insights on production AI logs, and how these can be used to drive daily workflow around fixing or improving agents in production.

Minute 14: Overview of a framework that AI product teams can use to implement analytics on production AI logs, including a fully free and open stack supporting this process.

Minute 20: Deeper explanation of the data pipeline and processing required to produce useful analytics on production AI logs that fully supports tab-complete analytics.

Minute 23: Architecture for stack used in this example, and explanation of how you can reproduce with a fully open and free set of tools.

Minute 24: Comparison of online analytics with offline evals, and how they complement each other as part of a robust observability stack.

Minute 35: Summary of example case studies and types of use cases where analytics on production AI have proven to be particularly valuable.

Minute 36: Description of issues that have arisen that would have been hard to catch without this type of analytics to provide intuition on how teams should use this product.

‍

Get started

Distributional is a free, open, and installable platform for agent analytics. Try it today and quickly learn how it complements your existing agent observability stack. We are also always happy to learn more about your use case and enterprise needs, so reach out to contact@distributional.com with any questions.