Program

MLOps is largely mature: the components are known, integrations are repeatable, and models can move to production in weeks rather than a year. LLM-based systems are not there yet, and demand for them is higher than anything I have seen.

The pressure comes from both directions. Leadership wants something agentic in production, while agentic coding means developers can build POCs in days and arrive expecting to deploy them. What they bring is often unlike anything the organization has operated before: new tools, integrations, data sources, permissions, memory, and execution patterns. And every application is different.

This puts platform teams in an uncomfortable position. They are being asked to productionize systems faster than reusable LLMOps patterns can emerge. The equivalent of a mature MLOps platform will take time to develop. The question is what platform teams should do in the meantime.

Your dashboards are green. Your API returns 200 OK. Your AI is quietly failing.

AI systems fail semantically, not just at the infrastructure and availability layer. A RAG pipeline can return stale context that elevates hallucinations. An agent can loop and burn tokens. A vendor can silently swap a model. Every health check stays green. The usual SRE practices were not designed to catch these failures.

This talk introduces a five-layer reliability model — foundation model, retrieval/RAG, agentic/tooling, product/experience, and cross-cutting — each with its own failure modes that cascade across layers. It presents a practical starting point built around three core SLIs: semantic quality, retrieval fidelity, and safety, and walks through a maturity ladder from no observability through to adaptive optimisation.

AISRE doesn't mean using AI to improve operations. It means applying SRE principles to AI products.

Every Monday at midnight, our forecasting pipelines kick off. By the time we get to the office, it's either a good week or a bloodbath.

We prefer good weeks. This talk is about how we engineered for more of them — scrappily, in Python, as a tiny data team without a dedicated SRE or a long line of specialized vendor tools.

It all started with an incident: some scheduled jobs stopped running, and nothing we'd built at the time was designed to notice. We were monitoring loud failures; silence was sorely missing from the list.

Observability (is the system behaving?) and explainability (why is it producing these numbers?) usually get treated as separate disciplines with separate stacks, but a small team often doesn't have the capacity to compartmentalize. So we treat it as one problem: is this forecast trustworthy enough to act on, and if not, how fast can we find out why?

I'll show what building instrumentation around this question looks like in practice, and what changed when we added agentic components to the mix. A system that can act on its own is also a system that can be confidently wrong on its own — and suddenly there's much more to watch, and exactly as many of us watching.

The system will keep finding new and exciting ways to ruin our Mondays — it just doesn't get to pull the same shenanigans twice.

The cost of software has always been defined by the cost of its maintenance. The safest, cheapest way to change software has historically been to issue a steady drip of small diffs against a baseline assumed to be good — the durable code model. This pattern has been so dominant in our lifetimes that it almost seems like a law of physics. But it is not. It is a model grounded in the economics of code production, in a world where producing working code is slow, hard and painstaking work.

That model has been upended in the last year, since AI can now generate code approximately as good as that of the median software engineer. Does this mean all software can and should be replaced by disposable code? What invisible tradeoffs have we been making? What lessons should we take from the immutable infrastructure revolution of the last twenty years, and how should they guide us today?

Like many companies, we moved quickly into GenAI with ambitious prototypes, new models, guardrails, and growing confidence. But in production, legitimate requests were blocked, prompts were exposed, test datasets failed to reflect real users, and architecture changes did not solve the underlying quality problems.

This talk shares what those failures were and how they recently pushed us to start adopting an evaluation-driven culture, where quality is defined early, measured continuously, and used to guide development, releases, and production decisions.

Software engineers are good at fixing things. When something breaks, we respond quickly, mitigate the impact, and move on. But the real value isn't in the immediate response; it's what we learn afterwards from it. Learning From Incidents (LFI) turns failures, near misses, and blind spots into shared knowledge. When the LFI process works, it prevents repeat issues and improves system resiliency. However, in practice, it's often rushed, inconsistent, or skipped entirely.

"It looks like you're writing a postmortem. Want help actually learning something from it?"

That's the idea behind the LFI Wizard — a guided assistant that asks better questions and helps turn raw data into meaningful insights. It pulls in relevant context and guides the user through targeted prompts so they can focus on what actually matters.

Here's the twist: I'm not a software engineer. My background is in the arts, and I've always been more comfortable with conversation than code. Building this tool with an AI coding tool showed me that this doesn't have to be a limitation — it's an advantage. In this talk, I'll share how I went from idea to a working solution, along with practical patterns, common pitfalls, and a simple mindset shift: treating AI tools less like magic, and more like a new team member you must set up for success.

In this talk, I'll discuss Herogen, Delivery Hero's autonomous software delivery agent, the impact of this technology, and what it implies for software engineering.

Can you throw an LLM at a production incident and expect useful results? A candid look from someone who runs a distributed AI system and reaches for Claude before reaching for a dashboard. Surprises, failures, and why the answer matters for every engineer carrying a pager.

AI-native applications produce data with a different shape than the systems we usually deploy. A single agent invocation can carry megabyte-sized prompts, hundreds of nested LLM calls, and metadata that customers want to query directly, often as arbitrary SQL.

That combination breaks a lot of assumptions. Kafka was sized for small messages. Node.js services serialize JSON on the main event loop and stall on multi-megabyte payloads. Postgres performs well when you can optimize a schema for known access patterns, which you can't when the queries are being written by an agent.

At Langfuse we've been building observability infrastructure for AI applications and have run into each of these as we scaled. In this talk I'll go through what we've learned: where the practical limits are in Kafka, Postgres, Node.js, and S3; how we ended up using object storage as the system of record with a separate database for queryable metadata; how "an agent will write arbitrary SQL against this" changes schema design, performance work, and isolation; and the S3 access patterns that keep costs predictable at billions of events.

The goal isn't to prescribe a stack. It's to give a clearer picture of the failure modes ahead and the design space for solving them.

There's no doubt LLMs can generate plausible, working code — but can we trust them to review, let alone approve it? An LLM's innate desire to be a "helpful assistant" backfires in code reviews. What should be a quick Pull Request check turns into a lavish, flattering essay praising test coverage and architectural genius of the author. Engineers tune out the noise, and critical logic bugs get buried under a pile of minor nitpicks.

This is the story of how we built an LLM code review and approval bot designed to act like a pragmatic senior engineer. Our goal was to create incentives in our socio-technical system to fight AI slop, but the second-order effects hit hard. In this session, I'll walk you through our failed initial attempts, how we managed to build developer trust, and how our evals didn't save us from our first major production incident.

AI can generate code faster than humans can write tests. More than speed though, when AI writes both the source and the test, it introduces a dangerous blind spot. Bugs that occur in the source code could also affect the test suite. The risk of bias and hallucinations in AI generated source and tests mean that there is no longer a last line of defense that prevents the end user from a sub-par experience.

Hand-written example-based tests cannot keep up with AI generated code. Property-based testing (PBT) offers a different solution that relies on deterministic verification of a problem space. Instead of enumerating individual test cases, PBT explores the invariants in a system. It tests the code by systematically trying to find inputs that could break the invariant.

The key idea here is that the exploration is deterministic, backed by mathematical probing of the problem space. We will walk through how PBT fights the logical holes of probabilistic source code and how it fits directly with spec-driven development.