Workshops

AI systems are notoriously hard to evaluate: They’re non-deterministic, quality is often subjective, and a pipeline with multiple steps like retrieval, prompts, and tool calls can fail silently. After coaching hundreds of engineers into AI, the skill I see missing most often is structured evaluation. Teams that are newer to AI tend to rely on spot-checks and vibes, and don’t know there’s a problem until a customer complains or something embarrassing ends up on Twitter. This workshop aims to fix that. We’ll cover evaluation from absolute beginner to production: picking metrics, building your first eval suite, component-level evals for everything from RAG to tool use, using LLMs as judges effectively, and evaluating multi-agent systems in production. You’ll write real evals during the session and leave with systems you can reuse at work and a way to feel confident in what you’re building.