Five Claude Sessions and a Thread Dump

The previous post explains what broke: a lock convoy inside Kafka’s purgatory, selected by leadership concentration, amplified by a producer that awaited one message at a time. This post is about how we found it. Most of the finding was done by five Claude Code sessions working in parallel against metrics, logs and source, with people in a war room holding the levers. It took about a working day from “we suspect a software regression” to a confirmed mechanism, and another three hours to prove it with a controlled experiment. Eight days of failures had preceded that day.

I am writing this down because the method mattered more than the model. The same tools, pointed at the same data without the discipline described here, would have produced a confident wrong answer by lunch. Twice it nearly did.

What this post is not. It is not an argument that an AI agent found the root cause. People chose the questions, took the thread dumps, killed the consumers and decided what to ship. The sessions gathered, computed and cross-checked at a pace and breadth a handful of engineers could not match while also running the incident. That division of labour is the point.

Where it started, and the first dead end

The first session got a one-paragraph brief: find the component that writes transactions into the analytics platform’s Kafka, use the systematic-debugging skill to work out what is wrong, and check a month of commit history for anything that could explain it, with read access to the repository.

It did that well. It located the producer class, walked the produce path, reviewed every commit since early August, found no regression, and correctly described the per-batch cancellation token that hides the real error. Then it hit a wall that is worth naming. The session had been given a goal, “debug why the producer is failing”, and a hook that would not let it stop until that goal was met. The honest answer at that point was “the code has not changed; the cause is on the broker side or in the network, and I need telemetry I do not have access to.” A goal phrased as a definitive root cause cannot be satisfied by an honest partial answer, so the session kept asking for data, the user kept not having it to hand, and both wasted an hour.

The lesson: give an agent a goal it can close with the evidence it has, or give it the evidence. “Narrow the cause and tell me what you would need next” is a goal. “Find the root cause” is a wish.

The setup that worked: one session per evidence stream

The second attempt changed the shape of the work rather than the prompt. Five Claude Code sessions ran side by side on one machine, each with a different set of tools and a different job:

Session	Tools	Job
Coordinator	none of the data connectors; the knowledge vault	Owns the synthesis. Briefs the others with precise questions (UTC windows, broker identities, metric names), merges their replies into one running note, talks to the war room.
Metrics	Prometheus (MCP)	Broker and node telemetry: request queues, listener idle ratios, latency components, leadership, purgatory sizes.
Broker logs	Elasticsearch (MCP), platform side	Kafka broker, operator and Kubernetes event logs.
Application logs	Elasticsearch (MCP), application side	The producer’s own logs and failure records. This cluster was under stress: one narrow query at a time, never in parallel.
Code review	GitLab (MCP)	Read the producer, the queue consumer and the client wrapper; produce a severity-ranked findings document; later, review the fix.

Two design choices did most of the work.

The coordinator had no data tools. It could not run a query to “just check”. It had to formulate a question precisely enough for a peer to answer, which forced every question to name a time window, a component and a metric. That constraint produced better questions than any prompt engineering would have. It also kept one place responsible for the narrative, so contradictions between sources surfaced as contradictions rather than being quietly averaged away.

Each specialist owned one source and wrote as it went. Every session kept a findings file with the exact query, its execution time, and the result, appended per pass. By the end of the day those files were the post-mortem’s evidence section, not a reconstruction of it. When the coordinator wrote “this broker was blocked from 13:18”, the sentence pointed at a query that anyone could re-run.

The day, in passes

The sessions did not work through a plan. They worked in passes: the coordinator asked a batch of questions, each specialist answered from its source, the answers either agreed or did not, and the next batch of questions came from the disagreements. Roughly:

Pass 1, morning: is it the producer? The application-logs session established the failure signature from the producer’s own logs: every batch failing at exactly its timeout budget, with a generic “cancellation token triggered” message and never a Kafka error, from one data centre only, since a precise minute eight days earlier. The code-review session explained why the real error could not appear in those logs. The metrics session showed broker-measured produce latency had never exceeded a few seconds. First contradiction: the client waited minutes, the broker measured seconds.

Pass 2: where does the time go? The metrics session found the gap: request queues at their cap on one or two brokers at a time, and per-processor idle ratios showing exactly one listener’s network threads at zero idle, which means blocked on the queue, not busy. Time in an unread socket is invisible to broker latency metrics. That reframed the incident from “slow broker” to “starved broker”, and from “the producer is broken” to “the producer is the loudest victim”.

Pass 3: what starves it? This is where two plausible wrong answers appeared and were killed. The first was a co-located data-processing workload bursting CPU on the broker nodes; it lined up with two evening episodes and not with the longest one, where a broker blocked for ninety minutes with no CPU, disk or reassignment pressure at all. The metrics session demoted it from cause to trigger in writing, with the counter-example. The second was a hypothesis raised in the war room that partitions had been added to a family of topics a month earlier and picked up by a wildcard subscription. The broker-logs session showed the relevant retention window did not reach that far back, that the consumers in question subscribed to none of those topics, and that their fetch width had changed by 0.3 percent in the retained period. Refuted, with the numbers, within twenty minutes of the suggestion.

Pass 4: predict the park site. By early afternoon the metrics pointed at a specific shape: handler threads consumed by follower fetches at 25 to 70 times their normal service time, with the follower fetch rate collapsing, which is lock wait, not work. The metrics session named the eight replication consumers as the likely lock holder by computing, for each broker, how many of the partitions those consumers read it led. The ranking matched the stall ranking exactly. Then the coordinator asked the war room for three thread dumps of the blocked broker, ten seconds apart, and told them which frames to expect.

Pass 5: the dumps. About thirty of thirty-six handler threads were parked on one lock object in each dump, in the frame the metrics had predicted. The code-review session was pointed at the Kafka source for the two versions and came back with the change in lock semantics between 2.6 and 2.7. The mechanism was now a chain of specific, checkable claims.

Pass 6, evening: the experiment. The platform team stopped the eight consumers. Within four minutes three sessions reported the effect from three sources: the broker-logs session saw the groups go empty by heartbeat expiry and the one churning broker fall silent; the metrics session saw purgatories collapse and throughput nearly triple with zero produce errors; the application side saw batch latency fall five-fold. An hour earlier the same sessions had measured a producer scale-up and reported, to some disappointment in the room, that it had changed nothing. Both results were in writing before the call ended.

What the sessions got wrong, and how the process caught it

A list of retractions is the most useful part of any account of AI-assisted work, so here is ours.

Claim	Who raised it	What killed it	Time to retraction
The producer had regressed in a recent commit	The original brief	A month of commits reviewed; the produce path unchanged	1 hour
A co-located workload’s CPU bursts were the cause	Metrics session	A ninety-minute block on a broker with no CPU pressure	2 hours; demoted to trigger, kept
A second producer fleet’s traffic had “collapsed”	War-room screenshot	Pre-incident baselines: the drop was a replay job finishing	40 minutes
Partitions were added a month earlier and picked up by wildcard	War room	Retention window; consumers subscribe to none of those topics; width changed 0.3 %	20 minutes
The replication consumers had been restarted or changed	War room	All eight groups present with the same width in the first retained metrics sample	30 minutes
A stalled broker’s request queue was fine	Metrics session, first read	The queue metric on combined broker-controller pods always reads zero because of a JMX bean collision	1 hour
Zero hits for a configuration line meant it was never logged	Broker-logs session	The line exceeds 256 characters, so its keyword field is dropped; a match query found it in one attempt	5 minutes

Three things made those retractions cheap. Every claim carried its evidence, so a counter-example had something concrete to hit. Every source had a second source next to it, so a wrong read in metrics was contradicted by logs or the client, usually within the hour. And retractions were written into the same running note as the claims, dated, so the war room could see what had been believed at 13:00 and why it was no longer believed at 14:00. Nobody had to remember.

The two “metric lied” rows deserve emphasis, because they are where an agent working alone would have built a confident case on a false floor. The queue metric that reads zero on half the brokers, the handler-idle metric exported as a lifetime mean, the container CPU counter that double-counts against the node exporter, the log field silently dropped above a length limit, the search analyzer that splits one hostname into three tokens. None of these is exotic. All of them produce clean-looking results that mean the opposite of what they appear to mean. The sessions caught them because the discipline was “an absence of hits is a claim about the query, not about the world” and because each trap, once found, was written into a shared memory that the other sessions read before their next query. By the afternoon nobody was rediscovering the tokenizer.

Rules we ended up with

These are the operating rules that emerged, in the order we would set them up again.

1. One source, one session, one findings file. An agent that can read everything will read the convenient thing. Constraint is what produces triangulation.
2. A coordinator with no data access. It has to ask, and asking forces precision. It also gives the humans one conversation to follow.
3. Every answer names its evidence. The query, the window, the execution time, the raw number. “The broker was blocked” is not an answer; “processors 24 to 31 at idle 0.00 from 13:18 to 15:49, query took 0.6 s” is.
4. Hypotheses are written down to be killed. Each one gets the observation that would refute it, and the session that owns that observation goes and looks. Retractions are written next to the claims they replace.
5. No hits is a hypothesis about the query. Before concluding an event did not happen, prove the query can find a known event of that kind.
6. Stressed systems get sequential, narrow, aggregation-only queries. The application-side log cluster was already struggling; the rule was one query at a time, a single day index, a cheap filter, size zero, and never retry a slow query, narrow it instead. The sessions followed it because it was stated once as a rule, not as a preference.
7. Predict before you look. The thread dumps were requested with the expected frames written down. A confirmation you predicted is worth ten you did not.
8. People hold the levers. Thread dumps, stopping consumers, deployments, resends. The sessions asked for them and measured them. They never did them.
9. Measure the negative results too. The producer scale-up that changed nothing was as important as the consumer switch-off that changed everything, because it ruled out the room’s favourite explanation before it hardened into folklore.
10. Write the post-mortem as you go. The findings files, the running note and the retraction table were the post-mortem. Writing it afterwards would have taken a week and lost the timestamps.

What the agents were actually good at

Not the insight. The mechanism was a textbook lock convoy, and the engineer who eventually saw the thread dump would have named it without help. What the sessions provided was breadth and stamina at the evidence layer:

• Reading eleven days of broker logs across a hundred million lines a day, with the right filters, in passes of six to twenty queries, each under two seconds, and reporting exact timestamps.
• Computing the per-broker count of partitions led among the consumers’ 4,742, and noticing that the ranking matched the stall ranking, which is the observation that turned “these consumers look suspicious” into “these consumers are the selector”.
• Holding thirty numbers steady across a six-hour conversation without rounding or drifting, and refusing to let a screenshot from the room override a query result.
• Reading two versions of Kafka’s DelayedOperation.scala and returning the exact semantic change, with the ticket number, in the time it takes to find the file.
• Reviewing a two-thousand-line producer path and producing eleven ranked findings, then reviewing the fix against those findings the next morning and catching the two it had missed.

And one thing they were good at that we did not expect: being unimpressed. When a mitigation was applied in the room and someone said it looked better, the sessions reported the numbers, which said it did not. When the switch-off worked, the sessions reported that it had, from three sources, and declined to call it confirmed until produce errors and messages-per-request ruled out retries as the explanation for the throughput jump. That temperament is not a model property. It is a consequence of rule 3.

What I would do differently

• Set up the parallel sessions on day one, not day six. The first five days were spent debugging the wrong layer because the evidence that would have redirected the investigation lived in a system the first session could not see.
• Give the coordinator a retraction table from the start, rather than growing one out of embarrassment.
• Put the metric traps in shared memory before the incident, not during it. Half of them are properties of the exporter, not of the incident, and were knowable in advance.
• Budget for the goal-hook problem. An agent told to find the root cause will keep looking for it after the honest answer is “I cannot, from here.” Phrase goals as questions the agent can close.

Closing

The incident took eight days because the failing component was the one with the best logs, and the component that caused it was the one with the worst. The investigation took one day because five narrow, disciplined readers were pointed at five sources at once, with one place where their answers had to agree. The model was the same in both cases. The difference was the method, and the method is reusable by anyone with the same tools: give each agent one source and one job, make it show its queries, make it write down what it stopped believing, and keep your own hands on the levers.

---

Companion to Anatomy of a Kafka Lock Convoy. Tooling named here, Kafka, Prometheus, Elasticsearch, New Relic, GitLab and Claude Code, is named because it is what we used; nothing about the organisation, its products or its customers is.