What’s broken in AI

On February 4, 2026, Scott Clark joined speakers from Tesla and Eventual at the San Francisco AI Engineering Meetup to discuss what is broken in AI with an audience of hundreds of bay area AI builders. You can watch the full talk here:

If you’d rather watch a 3-min summary of what’s broken in AI, you can view this video here:

Here is a run through of a few key points from Scott’s talk:

14:30 - From Optimization of ML to Analytics for AI: The problem is confidence, not performance. Teams worry more about agent behavior, not whether it will perform, and need insights into how they behave to gain the confidence needed to ship and scale them.

16:30: Analytics to Optimization: Analytics on production agent traces can be used to guide reinforcement learning, fine tuning, prompt optimization, context engineering, and other tasks to optimize agent performance – a new approach to objective function solicitation.

17:30: How Analytics Fits with AI Observability: Analytics trades off timeliness of monitoring for the richness of deeper insight on agent behavior, especially the unknown unknowns of usage patterns, agent behavior, and agent performance, and the correlation across these.

19:15: Hierarchy of Agent Observability: Logging to build and debug, monitoring to catch known issues quickly, and analytics to find issues and improve over time – especially unknown behaviors. This is the full stack of functionality needed for agent observability.

20:30: How Analytics Works: Discussion around analyzing traces from a demo outing agent to uncover trends, correlations, and clusters of interesting usage patterns and agent behavior – behavioral signals that drive insights on what to fix and improve.

22:30: Free and Open Distribution: Get started with the sandbox in minutes at docs.dbnl.com or install the full service for free.

Get started

If you have any of these problems, the easiest way to get started is to use a free SaaS demo account to review this example and other examples that we’ve pre-loaded in Distributional. Next, you can install our sandbox locally on your laptop in ten minutes and run through a tutorial that shows you how to use Distributional for a toy example. Once more familiar with our functionality, you can install the full service for free using a Terraform Module or Helm Chart. We are happy to help through any step of this process, so reach out at support@distributional.com with any questions.