Recorded live during the Lightning Talks at the MLOps World | GenAI Summit 2025 — Austin, TX (October 8, 2025) Session Title: Shipping AI That Works Speaker: Nick Luzio, AI Solutions Lead, Arize AI Talk Track: LLM Observability Abstract: When you’re deploying agents into real-world production systems, how do you know they’re actually working—and why they sometimes fail? In this lightning talk, Nick Luzio, AI Solutions Lead at Arize AI, breaks down the foundations of observability and evaluation for AI agents. He shares lessons learned from helping enterprise teams move beyond prototypes to production—covering key methods for debugging, validating, and continuously improving agent performance at scale. Nick walks through practical strategies for monitoring reliability and ensuring trust in agentic systems, highlighting how Arize’s observability tools can help teams iterate faster while keeping models aligned and transparent. What you’ll learn: • How to ensure AI agents work reliably at scale • Why observability and evaluation are critical to trustworthy AI • Practical approaches to debugging and refining agents in production • Tools and frameworks that support continuous improvement cycles.