The Symptom You're Actually Looking At
You open the RabbitMQ management UI, click into a queue, and see something that doesn't make sense: Unacked is sitting at a number greater than zero, but the Consumers panel reads 0. You restart the node. The number doesn't move. Nobody on your team admits to ever attaching a consumer to that queue.
This is confusing because it feels like a contradiction. It isn't. RabbitMQ's unacked counter is not a mystery metric that appears out of nowhere — it only increments when a message is actually delivered to a channel, whether that's through basic.consume or a manual basic.get. If unacked is above zero, a delivery happened. The real question is where that channel went, and why the broker still thinks it's alive.
Rule Out the Red Herrings First
Before you assume you've found a broker bug, check three things, because these are the usual false alarms.
First, confirm you're reading the right number. Ready messages and unacked messages are different columns — people sometimes glance at the wrong one, especially on a dashboard with multiple queues stacked close together.
Second, check consumers at the connection level, not just the queue level. Run:
rabbitmqctl list_consumers
rabbitmqctl list_channels name connection pid consumer_count messages_unacknowledged
rabbitmqctl list_connections name peer_host peer_port state
A queue page can show 0 consumers currently registered while a channel somewhere still holds messages from a consumer that unsubscribed or crashed without acking. The queue's consumer count and the channel's delivery history aren't always the same story.
Third, don't trust a node restart to "fix" this unless the connection holding the unacked messages was actually terminated by that restart. If your consumer connects to a different node in the cluster, or if the client auto-reconnects fast enough that the broker never sees the TCP session die, the unacked messages survive the restart untouched.
The Actual Mechanism: Unacked Messages Belong to a Channel, Not the Queue
This is the part most explanations skip. An unacked message isn't just "pending in the queue waiting for someone to click a button." It's bound to a specific channel on a specific connection. The queue hands it off and keeps it off the ready list until one of three things happens: the channel acks it, the channel nacks/rejects it, or the channel (or its parent connection) closes.
That third condition is the trap. If the broker never detects that the connection is gone, it has no reason to requeue anything. The message just sits there, forever, looking exactly like what Anup described: unacked, with no visible active consumer.
Failure Mode 1: The Zombie TCP Connection
This is the most common cause by far. A consumer process crashes, gets OOM-killed, or the network between client and broker drops a packet silently — but the TCP socket on the broker side doesn't get a FIN or RST. Without AMQP heartbeats enabled, RabbitMQ has no fast way to notice the client is gone. It just waits on a socket read that will never return data, sometimes for hours, until the OS-level TCP keepalive (if configured at all) eventually gives up.
Meanwhile, every message that was delivered to that channel before it died stays unacked. From the management UI, it can look like there are no consumers, because the consumer process is dead — but the broker-side channel object representing it hasn't been cleaned up yet.
Failure Mode 2: Consumer Cancellation Without Draining Pending Deliveries
Some client libraries let you cancel a consumer subscription (basic.cancel) without automatically nacking messages that were already delivered but not yet acked. If your application logic cancels a consumer mid-processing — say, during a graceful shutdown that doesn't wait for in-flight acks — those messages are orphaned on that channel. The consumer is gone from the registration list, but the delivered-and-unacked messages remain stuck until the channel itself closes.
Failure Mode 3: Debugging Through the Management UI or HTTP API
This one catches people during incident response, ironically while they're trying to diagnose something else. The management UI's "Get messages" feature and the HTTP API's GET /queues/.../get endpoint let you pull messages off a queue to inspect them. If you choose manual ack mode instead of "ack message / requeue true" and then forget to acknowledge, that message is now unacked, tied to a short-lived HTTP channel that won't show up looking like a normal long-running consumer. It's a self-inflicted version of the exact symptom in this article.
Failure Mode 4: Classic Mirrored Queue Edge Cases
On classic mirrored queues, a partition or mirror promotion during a delivery window can occasionally leave unacked state that looks orphaned from the new master's perspective until the old connection is fully reaped. This is one of the reasons the RabbitMQ team has been steering people toward quorum queues, which handle this kind of failover far more predictably. If you're choosing a queue architecture from scratch, it's worth comparing priority queue patterns and multiple-queue designs before you commit to one at scale.
The Fix: Find and Kill the Connection Holding the Messages
Don't restart the whole node again — go find the specific connection.
- List channels with nonzero
messages_unacknowledged:
rabbitmqctl list_channels pid connection messages_unacknowledged consumer_count
- Cross-reference the connection PID against
rabbitmqctl list_connectionsto get client details — peer address, client properties, connected_at. A connection that's been "connected" for days with no matching active process on the client side is your suspect. - Force-close it, either via the management UI's "force close" button on the Connections tab, or via CLI:
rabbitmqctl close_connection "<connection-pid>" "clearing orphaned unacked messages"
- Closing the connection tells the broker the channel is gone. Every unacked message tied to that channel gets requeued and moves back to Ready immediately.
If you genuinely find zero connections and the unacked count still won't move, that's unusual enough to warrant a packet capture, exactly as suggested in the original RabbitMQ mailing list thread this scenario comes from — an AMQP-level trace will show you whether some other process (a shovel, a federation link, a plugin) is quietly consuming and failing to ack.
Preventing This From Happening Again
A few configuration changes stop this failure mode from recurring instead of just fixing it once.
- Keep heartbeats enabled. Don't disable them for "performance," they're how the broker detects dead TCP sessions quickly instead of waiting on OS-level keepalive timers that can take a very long time.
- Set
consumer_timeout. Recent RabbitMQ versions support a consumer timeout that will forcibly close a channel if a delivered message isn't acked within a configured window, converting a silent zombie into a logged error and an automatic requeue. - Use quorum queues with a delivery limit where message ordering guarantees allow it. A delivery limit dead-letters a poison message instead of letting it loop between redelivery and an unresponsive consumer indefinitely.
- Alert on the mismatch itself. If you're running Prometheus and Grafana against your RabbitMQ metrics, add an alert rule for
messages_unacknowledged > 0 AND consumer_count == 0sustained for more than a few minutes. That exact combination is the signature of this issue, and catching it early means finding one stale connection instead of debugging a backlog days later.
Verifying the Fix Worked
After closing the suspect connection, check the queue page again. Unacked should drop and Ready should climb by the same amount. If you have a real consumer attached and running with manual ack, it should start pulling those messages normally within one poll cycle. If the count drops to zero and stays there without a new consumer ever attaching, that confirms the messages were sitting on a dead channel the whole time, not stuck due to some deeper queue corruption.
FAQ
Can unacked messages be lost if the broker crashes before I fix this? If the queue and messages are durable and the messages were published with the persistent delivery mode, no — they survive a broker restart and go back to Ready once the dead channel is detected or force-closed. If the queue is non-durable or the messages were published as transient, a hard crash can lose them regardless of ack state, which is a separate durability concern from this issue.
Why didn't restarting the node clear the unacked count in the original report? A restart only forces requeue if the connection holding the channel was actually torn down by that restart. In a cluster, if the client's connection lands on a node that wasn't restarted, or if the client reconnects fast enough that the broker never registers the old connection as dead, the unacked state persists right through the restart.
Is there a way to see which specific messages are stuck unacked, not just the count?
Not directly through acks alone — RabbitMQ doesn't expose per-message ack state in the management UI. The practical approach is identifying the channel via list_channels, closing its connection to force the requeue, and then inspecting the messages once they're back in Ready, using the "Get messages" feature with requeue enabled so you don't create the same problem again.