alex.bularca.me

Five Claude Sessions and a Thread Dump

Alex Bularca — Sat, 12 Sep 2026 00:00:00 GMT

The previous post explains what broke: a lock convoy inside Kafka's purgatory, selected by leadership concentration, amplified by a producer that awaited one message at a time. This post is about how we found it. Most of the finding was done by five Claude Code sessions working in parallel against metrics, logs and source, with people in a war room holding the levers. It took about a working day from "we suspect a software regression" to a confirmed mechanism, and another three hours to prove it with a controlled experiment. Eight days of failures had preceded that day.

I am writing this down because the method mattered more than the model. The same tools, pointed at the same data without the discipline described here, would have produced a confident wrong answer by lunch. Twice it nearly did.

What this post is not. It is not an argument that an AI agent found the root cause. People chose the questions, took the thread dumps, killed the consumers and decided what to ship. The sessions gathered, computed and cross-checked at a pace and breadth a handful of engineers could not match while also running the incident. That division of labour is the point.

Where it started, and the first dead end

The first session got a one-paragraph brief: find the component that writes transactions into the analytics platform's Kafka, use the systematic-debugging skill to work out what is wrong, and check a month of commit history for anything that could explain it, with read access to the repository.

It did that well. It located the producer class, walked the produce path, reviewed every commit since early August, found no regression, and correctly described the per-batch cancellation token that hides the real error. Then it hit a wall that is worth naming. The session had been given a goal, "debug why the producer is failing", and a hook that would not let it stop until that goal was met. The honest answer at that point was "the code has not changed; the cause is on the broker side or in the network, and I need telemetry I do not have access to." A goal phrased as a definitive root cause cannot be satisfied by an honest partial answer, so the session kept asking for data, the user kept not having it to hand, and both wasted an hour.

The lesson: give an agent a goal it can close with the evidence it has, or give it the evidence. "Narrow the cause and tell me what you would need next" is a goal. "Find the root cause" is a wish.

The setup that worked: one session per evidence stream

The second attempt changed the shape of the work rather than the prompt. Five Claude Code sessions ran side by side on one machine, each with a different set of tools and a different job:

Session	Tools	Job
Coordinator	none of the data connectors; the knowledge vault	Owns the synthesis. Briefs the others with precise questions (UTC windows, broker identities, metric names), merges their replies into one running note, talks to the war room.
Metrics	Prometheus (MCP)	Broker and node telemetry: request queues, listener idle ratios, latency components, leadership, purgatory sizes.
Broker logs	Elasticsearch (MCP), platform side	Kafka broker, operator and Kubernetes event logs.
Application logs	Elasticsearch (MCP), application side	The producer's own logs and failure records. This cluster was under stress: one narrow query at a time, never in parallel.
Code review	GitLab (MCP)	Read the producer, the queue consumer and the client wrapper; produce a severity-ranked findings document; later, review the fix.

Two design choices did most of the work.

The coordinator had no data tools. It could not run a query to "just check". It had to formulate a question precisely enough for a peer to answer, which forced every question to name a time window, a component and a metric. That constraint produced better questions than any prompt engineering would have. It also kept one place responsible for the narrative, so contradictions between sources surfaced as contradictions rather than being quietly averaged away.

Each specialist owned one source and wrote as it went. Every session kept a findings file with the exact query, its execution time, and the result, appended per pass. By the end of the day those files were the post-mortem's evidence section, not a reconstruction of it. When the coordinator wrote "this broker was blocked from 13:18", the sentence pointed at a query that anyone could re-run.

Coordinator session no data tools · owns the running note briefs peers, merges replies writes to the knowledge vault Metrics session Prometheus · queues, idle, latency, leadership, purgatory Broker-logs session Elasticsearch · broker, operator, Kubernetes events Application-logs session Elasticsearch · producer logs stressed cluster: one query at a time Code-review session GitLab · producer, consumer, client findings F1–F11, then the fix review questions ↔ evidence questions ↔ evidence War room (people) thread dumps · stop consumers · deploy · decide Knowledge vault running note · tool traps · recipes

The shape of the investigation. Four sessions each owned one data source and one findings file. The coordinator owned the story and could not run queries itself. People owned every action that changed production.

The day, in passes

The sessions did not work through a plan. They worked in passes: the coordinator asked a batch of questions, each specialist answered from its source, the answers either agreed or did not, and the next batch of questions came from the disagreements. Roughly:

Pass 1, morning: is it the producer? The application-logs session established the failure signature from the producer's own logs: every batch failing at exactly its timeout budget, with a generic "cancellation token triggered" message and never a Kafka error, from one data centre only, since a precise minute eight days earlier. The code-review session explained why the real error could not appear in those logs. The metrics session showed broker-measured produce latency had never exceeded a few seconds. First contradiction: the client waited minutes, the broker measured seconds.

Pass 2: where does the time go? The metrics session found the gap: request queues at their cap on one or two brokers at a time, and per-processor idle ratios showing exactly one listener's network threads at zero idle, which means blocked on the queue, not busy. Time in an unread socket is invisible to broker latency metrics. That reframed the incident from "slow broker" to "starved broker", and from "the producer is broken" to "the producer is the loudest victim".

Pass 3: what starves it? This is where two plausible wrong answers appeared and were killed. The first was a co-located data-processing workload bursting CPU on the broker nodes; it lined up with two evening episodes and not with the longest one, where a broker blocked for ninety minutes with no CPU, disk or reassignment pressure at all. The metrics session demoted it from cause to trigger in writing, with the counter-example. The second was a hypothesis raised in the war room that partitions had been added to a family of topics a month earlier and picked up by a wildcard subscription. The broker-logs session showed the relevant retention window did not reach that far back, that the consumers in question subscribed to none of those topics, and that their fetch width had changed by 0.3 percent in the retained period. Refuted, with the numbers, within twenty minutes of the suggestion.

Pass 4: predict the park site. By early afternoon the metrics pointed at a specific shape: handler threads consumed by follower fetches at 25 to 70 times their normal service time, with the follower fetch rate collapsing, which is lock wait, not work. The metrics session named the eight replication consumers as the likely lock holder by computing, for each broker, how many of the partitions those consumers read it led. The ranking matched the stall ranking exactly. Then the coordinator asked the war room for three thread dumps of the blocked broker, ten seconds apart, and told them which frames to expect.

Pass 5: the dumps. About thirty of thirty-six handler threads were parked on one lock object in each dump, in the frame the metrics had predicted. The code-review session was pointed at the Kafka source for the two versions and came back with the change in lock semantics between 2.6 and 2.7. The mechanism was now a chain of specific, checkable claims.

Pass 6, evening: the experiment. The platform team stopped the eight consumers. Within four minutes three sessions reported the effect from three sources: the broker-logs session saw the groups go empty by heartbeat expiry and the one churning broker fall silent; the metrics session saw purgatories collapse and throughput nearly triple with zero produce errors; the application side saw batch latency fall five-fold. An hour earlier the same sessions had measured a producer scale-up and reported, to some disappointment in the room, that it had changed nothing. Both results were in writing before the call ended.

What the sessions got wrong, and how the process caught it

A list of retractions is the most useful part of any account of AI-assisted work, so here is ours.

Claim	Who raised it	What killed it	Time to retraction
The producer had regressed in a recent commit	The original brief	A month of commits reviewed; the produce path unchanged	1 hour
A co-located workload's CPU bursts were the cause	Metrics session	A ninety-minute block on a broker with no CPU pressure	2 hours; demoted to trigger, kept
A second producer fleet's traffic had "collapsed"	War-room screenshot	Pre-incident baselines: the drop was a replay job finishing	40 minutes
Partitions were added a month earlier and picked up by wildcard	War room	Retention window; consumers subscribe to none of those topics; width changed 0.3 %	20 minutes
The replication consumers had been restarted or changed	War room	All eight groups present with the same width in the first retained metrics sample	30 minutes
A stalled broker's request queue was fine	Metrics session, first read	The queue metric on combined broker-controller pods always reads zero because of a JMX bean collision	1 hour
Zero hits for a configuration line meant it was never logged	Broker-logs session	The line exceeds 256 characters, so its keyword field is dropped; a match query found it in one attempt	5 minutes

Three things made those retractions cheap. Every claim carried its evidence, so a counter-example had something concrete to hit. Every source had a second source next to it, so a wrong read in metrics was contradicted by logs or the client, usually within the hour. And retractions were written into the same running note as the claims, dated, so the war room could see what had been believed at 13:00 and why it was no longer believed at 14:00. Nobody had to remember.

The two "metric lied" rows deserve emphasis, because they are where an agent working alone would have built a confident case on a false floor. The queue metric that reads zero on half the brokers, the handler-idle metric exported as a lifetime mean, the container CPU counter that double-counts against the node exporter, the log field silently dropped above a length limit, the search analyzer that splits one hostname into three tokens. None of these is exotic. All of them produce clean-looking results that mean the opposite of what they appear to mean. The sessions caught them because the discipline was "an absence of hits is a claim about the query, not about the world" and because each trap, once found, was written into a shared memory that the other sessions read before their next query. By the afternoon nobody was rediscovering the tokenizer.

Rules we ended up with

These are the operating rules that emerged, in the order we would set them up again.

One source, one session, one findings file. An agent that can read everything will read the convenient thing. Constraint is what produces triangulation.
A coordinator with no data access. It has to ask, and asking forces precision. It also gives the humans one conversation to follow.
Every answer names its evidence. The query, the window, the execution time, the raw number. "The broker was blocked" is not an answer; "processors 24 to 31 at idle 0.00 from 13:18 to 15:49, query took 0.6 s" is.
Hypotheses are written down to be killed. Each one gets the observation that would refute it, and the session that owns that observation goes and looks. Retractions are written next to the claims they replace.
No hits is a hypothesis about the query. Before concluding an event did not happen, prove the query can find a known event of that kind.
Stressed systems get sequential, narrow, aggregation-only queries. The application-side log cluster was already struggling; the rule was one query at a time, a single day index, a cheap filter, size zero, and never retry a slow query, narrow it instead. The sessions followed it because it was stated once as a rule, not as a preference.
Predict before you look. The thread dumps were requested with the expected frames written down. A confirmation you predicted is worth ten you did not.
People hold the levers. Thread dumps, stopping consumers, deployments, resends. The sessions asked for them and measured them. They never did them.
Measure the negative results too. The producer scale-up that changed nothing was as important as the consumer switch-off that changed everything, because it ruled out the room's favourite explanation before it hardened into folklore.
Write the post-mortem as you go. The findings files, the running note and the retraction table were the post-mortem. Writing it afterwards would have taken a week and lost the timestamps.

What the agents were actually good at

Not the insight. The mechanism was a textbook lock convoy, and the engineer who eventually saw the thread dump would have named it without help. What the sessions provided was breadth and stamina at the evidence layer:

Reading eleven days of broker logs across a hundred million lines a day, with the right filters, in passes of six to twenty queries, each under two seconds, and reporting exact timestamps.
Computing the per-broker count of partitions led among the consumers' 4,742, and noticing that the ranking matched the stall ranking, which is the observation that turned "these consumers look suspicious" into "these consumers are the selector".
Holding thirty numbers steady across a six-hour conversation without rounding or drifting, and refusing to let a screenshot from the room override a query result.
Reading two versions of Kafka's DelayedOperation.scala and returning the exact semantic change, with the ticket number, in the time it takes to find the file.
Reviewing a two-thousand-line producer path and producing eleven ranked findings, then reviewing the fix against those findings the next morning and catching the two it had missed.

And one thing they were good at that we did not expect: being unimpressed. When a mitigation was applied in the room and someone said it looked better, the sessions reported the numbers, which said it did not. When the switch-off worked, the sessions reported that it had, from three sources, and declined to call it confirmed until produce errors and messages-per-request ruled out retries as the explanation for the throughput jump. That temperament is not a model property. It is a consequence of rule 3.

What I would do differently

Set up the parallel sessions on day one, not day six. The first five days were spent debugging the wrong layer because the evidence that would have redirected the investigation lived in a system the first session could not see.
Give the coordinator a retraction table from the start, rather than growing one out of embarrassment.
Put the metric traps in shared memory before the incident, not during it. Half of them are properties of the exporter, not of the incident, and were knowable in advance.
Budget for the goal-hook problem. An agent told to find the root cause will keep looking for it after the honest answer is "I cannot, from here." Phrase goals as questions the agent can close.

Closing

The incident took eight days because the failing component was the one with the best logs, and the component that caused it was the one with the worst. The investigation took one day because five narrow, disciplined readers were pointed at five sources at once, with one place where their answers had to agree. The model was the same in both cases. The difference was the method, and the method is reusable by anyone with the same tools: give each agent one source and one job, make it show its queries, make it write down what it stopped believing, and keep your own hands on the levers.

Companion to Anatomy of a Kafka Lock Convoy. Tooling named here, Kafka, Prometheus, Elasticsearch, New Relic, GitLab and Claude Code, is named because it is what we used; nothing about the organisation, its products or its customers is.

Anatomy of a Kafka Lock Convoy

Alex Bularca — Fri, 11 Sep 2026 00:00:00 GMT

How eight single-member consumers, each fetching 4,742 partitions in one request, brought a ten-broker Kafka cluster to a standstill one broker at a time, why it only started after an upgrade from the 2.x line, and why the brokers' own latency metrics never noticed.

TL;DR. A very wide consumer fetch parks in DelayedOperationPurgatory as a single DelayedFetch registered under every partition it covers. Every produce, follower fetch and consumer fetch that completes on any of those partitions calls checkAndComplete, which takes that one operation's ReentrantLock and runs a completion check proportional to the fetch width. Since Kafka 2.7 that lock acquisition blocks instead of trying and moving on. With 864 watched partitions on one broker, 30 of 36 request-handler threads queued behind one lock, the request queue hit its cap, and the network threads of the busiest listener stopped reading sockets. Broker-measured latency stayed low because time in an unread socket is not measured. A producer that awaited one message at a time under a per-batch timeout turned every slow minute into cancelled and re-sent batches, feeding the convoy. Stopping the eight consumers gave 2.8× throughput at a fifth of the latency within three minutes.

The setup

The system is an event pipeline for high-volume financial-style transactions. An application writes a transaction to its database and publishes an event to a message queue. A publishing service consumes that queue in small batches and produces each event into a Kafka topic with 105 partitions on a ten-broker cluster, Kafka 3.7.0 in KRaft mode, deployed with an operator on Kubernetes with two brokers per physical node. Downstream, several hundred consumer groups read the resulting topics: analytics, rule engines, exports, and a replication job that copies production topics into another environment.

Three details of that setup turned out to matter more than everything else combined:

The replication job was eight consumer groups, one member each, subscribed by wildcard to 285 topics and therefore 4,742 partitions. Each member's fetch to a given broker spanned every partition that broker led.
The publisher sent one message per produce request, awaited each delivery report before sending the next, and cancelled the entire batch on one shared timer.
The cluster had moved from a Kafka 2.x release to 3.7.0 three months earlier.

Application transaction committed Message queue ack per message Publisher 1 msg / request SASL_SSL Kafka 3.7 · KRaft 10 brokers · 2 per node ≈ 9k partitions · 1.5k topics Analytics consumers hundreds of groups Rule engines, exports latency-sensitive Replication job 8 groups, 1 member each 4,742 partitions per member

The pipeline. Everything in this post happens inside the brokers and in the publisher immediately in front of them. The replication job is an ordinary consumer; its shape is what made it dangerous.

The symptom, and why the metrics lied

From the publisher's point of view, produce calls started taking 30 to 240 seconds and failing. From the broker's point of view, nothing was wrong: Produce total time p99 never exceeded about 7.5 seconds during the worst episodes, and the request-handler idle ratio looked fine. For five days the incident was debugged as a client-side regression.

The gap between those two views is the first lesson of this incident, and it is structural. Kafka's request metrics start the clock when a network thread has read a complete request off the socket and enqueued it. If the network thread is not reading, the request is not in any metric. Meanwhile the handler idle metric on a broker with combined broker-and-controller roles is unreliable because of a JMX bean collision between the two request pools, so the pods that were actually stalling reported idle ratios above 1.0.

The metric that did show it, in every episode, was kafka.network:type=RequestMetrics,name=LocalTimeMs,request=FetchFollower: replica-fetch local time rising from 1 to 2 ms to 10 to 90 ms on one broker at a time, while the follower-fetch rate collapsed, handler CPU went down and the Produce purgatory size climbed. Followers touch every partition, so they hit the mechanism first. The second was per-processor NetworkProcessorAvgIdlePercent: the eight processors of the external TLS listener at zero idle while the internal listener's processors were fine.

Why "idle at zero" meant blocked, not busy. A busy network thread has a high io-ratio plus io-wait-ratio. A blocked one has both near zero: it is parked on a queue put, doing no I/O and not waiting on select either. On the stalled brokers the external listener's io-ratio + io-wait-ratio collapsed to about 0.1.

The mechanism

1. A convoy, not a deadlock

In a deadlock nothing moves. Here everything moved, through one door. One handler held a lock while it did a long piece of work. Every handler that needed the same lock parked behind it. When the holder released, the next one took the lock and did the same long work. The pool never stopped; it just spent almost all of its time queuing. Three SIGQUIT thread dumps of the stalled broker, taken about ten seconds apart, showed 33, 32 and 30 of the 36 data-plane handlers parked on the same lock object each time.

Healthy: many small locks Convoy: one lock, one door one lock per waiting operation, few threads per lock 36 handlers · 33 working, 3 waiting 36 handlers · 1 working, 30 parked, 5 free lock(M) one DelayedFetch, watched under 864 partition keys every partition event on the broker ends here

What the thread dumps showed. Handler CPU went down while the broker was "busy", because parked threads burn nothing. The controller-plane handler pool on the same JVM was idle throughout; only the data plane was involved.

2. How a fetch parks in purgatory

A consumer fetch that cannot be answered immediately, because fetch.min.bytes is not yet satisfied, becomes a DelayedFetch in the broker's DelayedOperationPurgatory. The purgatory must wake it the moment data arrives on any partition in the request, so it registers the same operation object in a Watchers list keyed by TopicPartition, once per partition in the fetch. A fetch covering 864 partitions on this broker is one object under 864 keys.

Whenever a handler finishes work on a partition, appending a produce, serving a follower fetch, or reading for a consumer, ReplicaManager calls purgatory.checkAndComplete(key) for that partition. That walks the partition's watch list and, for each operation, calls safeTryComplete(), which takes the operation's lock and runs tryComplete(). For a DelayedFetch, tryComplete iterates every partition in the fetch to compute accumulated bytes, and if the threshold is met it calls onComplete, which reads the log for every partition and builds the response, still under the same lock.

DelayedOperationPurgatory[DelayedFetch] on one broker appendRecords(p17) fetch (follower) p402 appendRecords(p88) fetch (consumer) p3 … any completion on 864 of the ≈ 990 partitions this broker leads checkAndComplete(key) Watchers, keyed by TopicPartition p17 → [ M ]p402 → [ M ]p88 → [ M ]p3 → [ M ] 864 keys, one object tryCompleteWatched → M.safeTryComplete() M: DelayedFetch lock = ReentrantLock fetchPartitionStatus: 864 entries consumer millions of msgs behind inLock(lock) { tryComplete(): for 864 partitions → accumulated bytes ≥ min? onComplete(): read logs, build the 864-part response } Cost held under the lock ∝ fetch width · Threads colliding ∝ watched partitions · Arrival rate ∝ produce + fetch rate on those partitions

One object, one lock, hundreds of doors into it. This is a correct and ordinary Kafka data structure. It assumes no single waiting operation is watched under hundreds of keys on one broker, and that its completion check is cheap. Both assumptions fail for a single-member consumer that fetches thousands of partitions while millions of messages behind.

Here is the reconstructed stack the parked handlers shared, abridged from the dumps:

"data-plane-kafka-request-handler-12" #… daemon prio=5 WAITING (parking)
  at jdk.internal.misc.Unsafe.park
  at java.util.concurrent.locks.LockSupport.park
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
  at java.util.concurrent.locks.ReentrantLock.lock
  at kafka.utils.CoreUtils$.inLock
  at kafka.server.DelayedOperation.safeTryComplete                 <-- 30 of 36 threads here
  at kafka.server.DelayedOperationPurgatory$Watchers.tryCompleteWatched
  at kafka.server.DelayedOperationPurgatory.checkAndComplete
  at kafka.server.ReplicaManager.$anonfun$addCompletePurgatoryAction / tryCompleteActions
  at kafka.server.ReplicaManager.appendRecords          // or fetchMessages, for the follower/consumer variants
  at kafka.server.KafkaApis.handleProduceRequest
  at kafka.server.KafkaRequestHandler.run

Thirty of these per dump, three dumps, three distinct lock objects across the ten-second span as one fetch completed and the next one from the same consumer parked. The single thread that was not waiting was inside DelayedFetch.tryComplete or onComplete.

3. From parked handlers to a frozen listener

Handlers are the only threads that dequeue from RequestChannel. With thirty of them parked, the queue reaches queued.max.requests (500 here) within minutes. Network processors block on the queue's put and stop calling select. Kafka assigns processors per listener, so the listener carrying the most requests fills the queue first and blocks first. On this cluster that was the external SASL_SSL listener the publishers used; in-cluster clients on the plaintext listener barely noticed.

SocketServer → RequestChannel → KafkaRequestHandlerPool, during the stall Client listener (TLS) 8 processors · idle 0.00 blocked in requestQueue.put() Internal listener (plaintext) 8 processors · idle 0.6–0.9 Replica listener 8 processors · slow RequestChannel.requestQueue 500 / 500 = queued.max.requests 36 data-plane handlers (num.io.threads) 30 parked in safeTryComplete · 5 free 1 inside DelayedFetch.tryComplete

Why the outside saw a frozen broker while the inside looked fine. Time a request spends unread in a socket is invisible to every RequestMetrics timer. The publishers waited 30 to 240 seconds; the broker's own Produce p99 stayed under a few seconds.

4. Why it started after the upgrade

The replication consumers had run in this shape for months on the 2.x cluster. What changed is how DelayedOperation behaves when a handler finds the lock already held. Abridged from the two sources:

// Kafka 2.5 / 2.6 — DelayedOperation.maybeTryComplete (abridged)
private[server] def maybeTryComplete(): Boolean = {
  var retry = false
  var done  = false
  do {
    if (lock.tryLock()) {                       // non-blocking
      try { tryCompletePending.set(false); done = tryComplete() }
      finally { lock.unlock() }
      retry = tryCompletePending.get()
    } else {
      // someone else holds it: leave them a note and move on
      retry = !tryCompletePending.getAndSet(true)
    }
  } while (!isCompleted && retry)
  done
}

// Kafka 2.7 → 3.7 — DelayedOperation.safeTryComplete (KAFKA-8334)
private[server] def safeTryComplete(): Boolean = inLock(lock)(tryComplete())   // blocks

KAFKA-8334 fixed a real bug: with the try-and-flag scheme, a completion could be missed when the flag was cleared at the wrong moment. The fix is correct. It also changes the cost model: a handler that used to skip a held lock now waits for it, and the wait includes the holder's tryComplete over every partition in the operation. For ordinary fetches of a few dozen partitions nobody notices. For one fetch of 864 partitions, watched under 864 keys, with a produce rate of thousands per second onto those partitions, the pool collapses into a queue.

5. The selector: which broker stalls

The convoy forms on whichever broker leads the most of the partitions the wide consumers read. Balanced across ten brokers, each leads about 470 of the 4,742, and the lock's arrival rate stays under what one thread can serve. The September hardware migration, a node retirement, several large reassignments and finally a storage-controller failure kept concentrating leadership on one broker at a time. The stall moved five times in eight days, and the ranking of replicated partitions led per broker predicted the ranking of stall severity exactly.

Partitions led per broker among the 4,742 the replication job reads · 10 Sep 13:30 0200400600800 balanced ≈ 470 864A 771B 756C 739D 674E 603F 258G 77H two further brokers were offline after a storage-controller failure when this sample was taken

Leadership is the selector. When two repaired brokers rejoined and a leadership-only reassignment equalised leaders to about 980 per broker, Broker A fell to about 470 replicated partitions and released within the next two-minute scrape. That is a margin, not a fix: any restart, node loss or rebalance re-arms it.

Blocked: client-listener processors idle 0.00, 30 handlers parked · Elevated: FetchFollower local p75 17–47 ms · Clean: ≤ 2 ms

6. The amplifier: the producer

A slow broker explains slow produces. The publisher's design explains why slow produces became eight days of failed batches and a growing backlog. Its produce loop, in shape:

// one timer for the whole batch, sequential single-message produces
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(batchBudget));   // 120 s, later 240 s
foreach (var msg in batch)
{
    await producer.ProduceAsync(topicPartition, msg)
                  .ContinueWith(r => RecordResult(r, msg.Id), cts.Token);   // token cancels the continuation
}

Four consequences follow directly:

Budget per batch, latency per message. A 60-message batch fails whenever 60 × per-produce latency > budget, even if every single produce succeeds. At 2 s each that is exactly 120 s.
The real error is unobservable. The token cancels the continuation, not the produce. The delivery report, and with it Local: Message timed out or the broker's error code, is never read. The log says only that a timer fired. Five days of debugging went into the wrong layer because of this.
Duplicates. The in-flight message stays in librdkafka's queue and may still be delivered; the batch is re-queued and the same message produced again. On the worst day, the ratio of queue deliveries to unique messages was about 1.74.
Load. Every failed batch is re-produced in full, one request per message, onto the very partitions under the convoy. The publisher alone accounted for about half the cluster's produce requests at 1.02 messages per request. librdkafka cannot coalesce what it never holds: with one outstanding message per loop, linger.ms has nothing to batch.

Broker slowslock convoy on its partitions Batch token firesN × latency > budget Continuation cancelleddelivery report never read Batch re-queued, re-sent in fullin-flight message duplicated more 1-message produce requests → more lock(M) acquisitions on the same partitions → broker slows further Observed: 75 minutes of continued batch failures after the brokers had fully recovered, purely from this path, on a backlog drain.

Two systems, one loop. The consumers made the brokers fragile; the producer converted fragility into failures, duplicates and extra load. Each side can break the loop independently.

The fix on the producer side is the standard shape, and it is worth spelling out because the first version of the rewrite kept two of the old bugs. Produce everything in the batch without awaiting each message, let each delivery handler complete its own TaskCompletionSource, await the batch once with a bounded wait, and decide per message from the delivery report:

var pending = batch.Select(msg =>
{
    var tcs = new TaskCompletionSource<DeliveryReport>(TaskCreationOptions.RunContinuationsAsynchronously);
    try
    {
        producer.Produce(topicPartition, msg, report => tcs.TrySetResult(report));   // non-blocking
    }
    catch (ProduceException<K, V> e)                                                // Local: Queue full, serializer errors
    {
        tcs.TrySetResult(DeliveryReport.Failed(e.Error));                          // per-message outcome, never abort the batch
    }
    return (msg, tcs.Task);
}).ToList();

var all = Task.WhenAll(pending.Select(p => p.Task));
if (await Task.WhenAny(all, Task.Delay(batchBudget)) != all)
{
    // budget exceeded: mark still-pending items failed with a *specific* reason;
    // do not cancel their continuations
}

foreach (var (msg, task) in pending)
    Ack(msg, task.IsCompletedSuccessfully && !task.Result.Error.IsError, task.Result?.Error);   // log Error.Code and PersistenceStatus

Two producer settings finish the job. message.timeout.ms should sit below the batch budget so librdkafka fails stragglers with a report before the application's timer does. And linger.ms has to be raised deliberately: at roughly 430 messages per second per host spread over ten brokers, the default 5 ms almost never coalesces, and the wire still carried 1.06 messages per request after the rewrite. A 100 to 200 ms linger buys a four- to nine-fold reduction in request count for a latency cost no analytics consumer will notice. Turn on enable.idempotence so that the retries that remain cannot duplicate.

The experiment

By the evening of the worst day, leadership had been re-balanced and no broker configuration had changed for two hours. That made stopping the eight replication consumers a clean test: if the convoy was the cause, the cluster should improve immediately, everywhere, and in real deliveries rather than retries.

Stopping the eight consumers, minute by minute (10 Sep, UTC) 010k20k30k40k 0100200300400 ms produce req/s produce p99 consumers stopped 17:41:14–17:42:55 17:4017:4117:4217:4317:4417:4517:4617:4717:4817:49 Bars: publisher produce requests/s (left). Line: cluster max Produce p99 (right). Errors 0 every minute · 1.02 msgs/request.

2.8× throughput at a fifth of the latency, within three minutes, with nothing else changed. Messages appended on the brokers tripled, so this is delivery, not retries. On the broker that had carried the early signature, follower-fetch local p75 went from 12 to 15 ms to 1 ms and its request queue from 500 to single digits in the same minute; Produce purgatory size collapsed on every broker.

The mirror side of the same evening is instructive too: raising the publisher's instance count from 11 to 20 and then to 30, an hour earlier, produced no measurable throughput and slightly higher broker latency. More senders into the same full queue. Ninety handlers behind one lock are still one lock.

Timeline, compressed

When	What
11 Jun	Cluster migrated from a 2.x release to 3.7.0. First broker-side lag incident two days later; two more in August, each "solved" by moving load off a slow node.
2 Sep 17:43	Hardware migration concentrates leadership on two freshly moved brokers. One replication consumer closes and reopens (new client id), issuing a fresh full fetch across all 4,742 partitions.
2 Sep 17:46	Publisher begins failing every batch at its 120 s budget. Onset.
3 – 7 Sep	1.6 – 1.85 M failed batch executions per day. Broker logs: not one WARN or ERROR during any stall. Consumers 2 – 40 h behind. A broker leaves for node retirement on 4 Sep; the cluster runs on nine.
5 Sep 08:00	The publisher's upstream queue is cleared during the incident. About 36 hours of events never reach Kafka and are later re-sent from the source database. No dead-letter path existed.
7 Sep 15:50	Leadership of the hot topic moved off the stalled broker; failures drop 15× that second. The convoy moves to the next most-loaded broker for two hours, then subsides. Evening episodes on 8 and 9 Sep.
10 Sep 07:16 – 12:30	Three multi-terabyte rebalances approved and stopped in turn; each landing of replicas on the most-loaded node triggers a short episode.
10 Sep 12:45	Storage-controller failure on one node: two brokers crash, 14 partitions offline, 140 under min-ISR, `NOT_ENOUGH_REPLICAS` at 440/s. Eight brokers left.
10 Sep 13:18	Broker A, now leading 35 % more partitions, blocks completely with no CPU, disk or reassignment pressure.
10 Sep 14:10	Three thread dumps. The lock holder is named; partitions led per broker match the stall ranking.
10 Sep 15:31 – 15:49	Repaired node returns; preferred-leader election, then a leadership-only reassignment equalises leaders. Broker A releases within two minutes.
10 Sep 16:48	Last failed batch, after a 75-minute tail of failures at exactly the 240 s budget with no Kafka error: the amplifier alone.
10 Sep 17:41	Eight replication consumers stopped. Three minutes later: 2.8× throughput, ⅕ latency, zero errors. First clean night.
11 Sep 08:06	Rewritten producer deployed: non-blocking sends, per-message delivery reports, real error logged. Batches four times larger, twice as fast, zero errors. Wire still 1.06 msg/request until linger is raised.

What I would tell another Kafka operator

Consumers

Budget partitions per consumer member, not per topic. Any group that would read more than a few hundred partitions per member must be sharded before it is deployed. Treat it like replication factor: a reviewed number.
Shard replication jobs across many members with the cooperative-sticky assignor and static membership (group.instance.id), so a member restart does not trigger a full rebalance and a fresh full-width fetch. With 16 members the widest fetch on any broker here would be about 55 keys.
Split wildcard subscriptions by topic family. A wildcard that silently grows with every new topic is how a fetch width becomes unknowable.
A consumer that is millions of messages behind should be completing fetches immediately. If it parks in purgatory at all, its fetch.min.bytes / fetch.max.wait.ms are wrong for its width. Cap fetch.max.bytes and max.partition.fetch.bytes so the response built under the lock is small.
Watch for consumer recreation. Two of the eight groups here recreated their consumer every one to two minutes; every recreation is a new fetch session and a full, non-incremental fetch, with the broker's session cache already full at max.incremental.fetch.session.cache.slots=1000.
Prefer a purpose-built replicator (MirrorMaker 2 with tasks.max per topic family) over a hand-rolled single-consumer copy job. Replication tools partition the work by design and expose lag and throughput per task.

Brokers

Request quotas for wide principals. A request_percentage quota on the replication principal would have throttled one badly shaped client instead of starving everyone behind the shared handler pool.
Keep leadership balanced automatically and alert on it: PreferredReplicaImbalanceCount > 100, or any broker leading more than 1.3× the average. Never leave a broker at 1,600 leaders for hours because the rebalancing tool's self-healing is broken.
Rack awareness (broker.rack = physical node) when running two brokers per host. One controller failure here turned into 14 offline partitions and 140 under min-ISR because both replicas of a partition shared a host.
Do not treat num.io.threads or queued.max.requests as the fix for a convoy. More threads behind one lock are more parked threads. Check release notes after 3.7 for purgatory changes before treating an upgrade as a fix, and know that a correct upgrade can change the cost of an existing pattern.
Isolate broker CPU from co-tenant workloads (requests, ideally Guaranteed QoS) and keep reassignments out of peak hours with two approvers. Several evening episodes here were timed by a co-located workload's CPU bursts and by rebalances approved during the incident.

Producers

Never await one message at a time under a batch-wide timeout. Produce with delivery handlers, await once, decide per message.
Always read the delivery report. The error you are not logging is the one you will spend five days looking for elsewhere.
Set message.timeout.ms below any application budget so librdkafka fails stragglers with a reportable reason before your own timer does.
Raise linger.ms deliberately and enable idempotence. At low per-connection rates the default linger coalesces nothing, and every retry without idempotence is a potential duplicate.
Retries are load. Back-pressure and pausing replay jobs when produce latency rises beat re-sending whole batches into a struggling broker.

Detection: PromQL that would have caught it in minutes

# Earliest and most specific: follower-fetch local time on any broker (normal 1–2 ms)
histogram_quantile(0.75, sum by (pod, le) (rate(kafka_network_requestmetrics_localtimems_bucket{request="FetchFollower"}[3m]))) > 10

# A blocked listener: its processors' idle collapses to ~0 while handler CPU also falls
min by (pod, listener) (kafka_network_processor_idle_percent) < 0.2

# Request queue at cap (unreliable on combined broker-controller pods: bean collision reports 0)
kafka_network_requestchannel_requestqueuesize >= 0.9 * 500

# Produce purgatory climbing on one pod — the companion signal in every episode
kafka_server_delayedoperationpurgatory_purgatorysize{delayedOperation="Produce"} > 400

# The selector: leadership concentration
kafka_controller_kafkacontroller_preferredreplicaimbalancecount > 100
max(kafka_server_replicamanager_leadercount) / avg(kafka_server_replicamanager_leadercount) > 1.3

# The lock holder's shape: partitions per consumer member (kafka_exporter)
count by (consumergroup) (kafka_consumergroup_lag) / on (consumergroup) sum by (consumergroup) (kafka_consumergroup_members) > 300

When the first of these fires, take three thread dumps ten seconds apart on the affected broker. If the data-plane handlers are parked in DelayedOperation.safeTryComplete under Watchers.tryCompleteWatched, it is this mechanism, and the immediate mitigation is to move leadership off that broker or pause the widest consumer.

Closing

Nothing in this incident was exotic. A replication job with an unusual but legal shape, a Kafka fix that closed a real bug, a hardware migration that moved leaders around, and a producer written the way most first Kafka producers are written. Individually, each had been fine for months. Together, with the lock semantics of Kafka 2.7 and later, they formed a convoy that no single dashboard showed. The two things I would want any team running Kafka to take from it are: partitions per consumer member is a capacity number, and read your delivery reports.

Mechanism confirmed from three thread dumps of the stalled broker, per-episode Prometheus metrics, broker logs, and the DelayedOperation.scala source for Kafka 2.5.0 and 3.7.0. Numbers are as measured at the time. Kafka internals described here are those of 3.7.0; check the release you run.