Reflections on AI day with NVIDIA

We recently had the pleasure of spending an inspiring day focused on AI with the NVIDIA team at their offices, including Jensen Huang. There were too many takeaways from the day, most of which I’m not at liberty to share. But here are a few I can. 

Product-led development (beyond post-training RL)

Since the rise of foundation models, AI teams implementing them have focused on post-training steps to maximize gains, including prompt iteration, context engineering, and reinforcement learning. Despite these efforts, teams that are scaling their AI applications are finding it hard to improve these AI products. They are sitting on mountains of traces, but are failing to find the signal in them. The people most responsible for scaling their AI products were the most interested in our product, thinking of us as a way to give them these signals to guide how they are evolving their products over time.

Value of deep tech

NVIDIA is world-class at developing libraries that optimize GPUs for a variety of tasks, workloads, and purposes. In the early days for startups finding product-market fit, it isn’t worth it to invest the energy in integrating these optimizations into your stack. Instead, you should be iterating on your core user experience to find value. But as you find product-market fit and start to scale, this deep tech adds significant value for a startup (or any company). In some computationally intensive cases, it can be the difference between healthy gross margin and break even. This is where NVIDIA’s team can bring a lot of value to partnership with AI startups.

Rapid consumer AI adoption, margin be damned

Although enterprises are lagging a bit, consumer adoption of AI continues to be happening at a mind blowing pace. In some cases, adoption is to generate cat memes, but other adoption feels much more tethered to real use cases that hold real, sustaining value. And popular belief among many in the industry is that it is worth sacrificing margin in the short term to gobble up as much market share as possible. There will be plenty of opportunity to adjust the token dial in the future—or leverage aforementioned deep tech—to generate healthy margins. 

AI platforms as a play

Although enterprise adoption is lagging behind consumer adoption, most of this seems to be stemming from the enterprise operating in a more complex environment and taking the time to get the initial set of applications right. The level of investment is there, and there is a lot of confidence that AI will make a meaningful impact on productivity writ large. To accommodate this trend, large system implementers and hyperscalers are investing in AI platforms—or factories—that abstract away some of this complexity from the enterprise, making it easier for them to rapidly adopt AI. Increasingly, these platforms are focusing on optimizing computation as a key value add in the stack they are delivering to customers, which is where optimized software from NVIDIA comes into play.

Vertical AI on the rise

Agentic-first vertical SaaS is on the rise in two ways. First, for existing software categories, startups that have built their entire stack around being agent first are making inroads and growing rapidly, such as Decagon and other customer experience applications. Second, an agent-first approach is making it possible to create entirely new software categories, such as Harvey and other legal applications. In both cases, the shift from horizontal to vertical seems to be happening faster than prior industry transitions, likely because the enabling technology (LLMs) is so powerful for these knowledge work use cases.

Transformers as enabling tech

Since the Attention paper, transformers have obviously dominated the AI architecture discussion for language. But what is less discussed is how enabling this architecture shift has been for other applications like vision as well. Use cases that were computationally intensive or not functionally feasible previously are now enabled with this new architecture, such as at scale analysis of video recordings for security breaches at corporate campuses. I suspect we will see a growing number of compelling use cases well beyond language as adoption of these architectures continues to grow across industries and applications.

What's next

These takeaways are only the tip of the iceberg of insights from this day. It was motivating to be surrounded by a passionate group of technologists living on the bleeding edge of this platform shift, and surfing the wave as it shifts in a new direction each day. I love trading notes on these sorts of thoughts. If you do too, reach out if you want to chat – nick-dbnl@distributional.com.