Declare RabbitMQ queues and exchange durable by default - #40
Open
nook24 wants to merge 1 commit into
Open
Conversation
A queue that is neither durable nor exclusive is RabbitMQ's deprecated
transient_nonexcl_queues feature. That is no longer only a deprecation:
RabbitMQ 4 reports it as denied_by_default and refuses every
queue.declare with a connection exception. Because Connect() returns
false as soon as a declare fails, RabbitMQ does not merely lose those
queues - it fails to connect at all, and publishes nothing.
Verified against RabbitMQ 4.3.5:
rabbitmqctl list_deprecated_features
transient_nonexcl_queues | denied_by_default | denied
Durable queues have worked since AMQP 0-9-1, so this is the only setting
that works on every RabbitMQ version, from 3.x through 4.x.
This does not put monitoring events on disk. Queue durability and message
persistence are separate AMQP properties: durable stores the queue
*definition*, while a message is only written durably when its publisher
marks it persistent. SendMessage passes properties=nullptr, so every
message this module publishes is transient and stays that way. The
RabbitMQ documentation is explicit that transient messages "will be
discarded during recovery, even if they were stored in durable queues".
So the behaviour Statusengine wants is unchanged: the queues buffer in
RAM while no worker is connected, and a RabbitMQ restart empties them.
Measured rather than assumed - 5 messages in a durable queue, RabbitMQ
restart, queue present with 0 messages. Publishing 20,000 events took
0.22-0.23s with durable queues and 0.22-0.23s without, three runs each.
The exchange follows the queues. A transient exchange loses its bindings
on a RabbitMQ restart while the durable queues survive, so the pair is kept
consistent; both are metadata only and neither costs per-message I/O.
Both values remain configurable, so an installation that needs the old
behaviour can still set DurableQueues = false - on a RabbitMQ old enough to
accept it.
Tested with Naemon 1.4.1 against RabbitMQ 3.9.27, publishing to all
configured queues, alongside the Go worker consuming them. Because both
sides declare the same queues and AMQP answers a mismatched redeclare
with a 406 PRECONDITION_FAILED rather than reconciling it, this change
belongs with the matching one in Statusengine Go Worker; both start orders
were checked, module first and worker first, with no 406 either way.
Note for existing installations: the queues already exist as non-durable
and cannot be redeclared. They have to be deleted once, with the
monitoring core and the worker stopped. Messages waiting in them are
lost, which is acceptable for the same reason the design is - they are
transient and would not have survived a broker restart either.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A queue that is neither durable nor exclusive is RabbitMQ's deprecated transient_nonexcl_queues feature. That is no longer only a deprecation: RabbitMQ 4 reports it as denied_by_default and refuses every queue.declare with a connection exception. Because Connect() returns false as soon as a declare fails, RabbitMQ does not merely lose those queues - it fails to connect at all, and publishes nothing.
Verified against RabbitMQ 4.3.5:
Durable queues have worked since AMQP 0-9-1, so this is the only setting that works on every RabbitMQ version, from 3.x through 4.x.
This does not put monitoring events on disk. Queue durability and message persistence are separate AMQP properties: durable stores the queue definition, while a message is only written durably when its publisher marks it persistent. SendMessage passes properties=nullptr, so every message this module publishes is transient and stays that way. The RabbitMQ documentation is explicit that transient messages "will be discarded during recovery, even if they were stored in durable queues".
So the behaviour Statusengine wants is unchanged: the queues buffer in RAM while no worker is connected, and a RabbitMQ restart empties them. Measured rather than assumed - 5 messages in a durable queue, RabbitMQ restart, queue present with 0 messages. Publishing 20,000 events took 0.22-0.23s with durable queues and 0.22-0.23s without, three runs each.
The exchange follows the queues. A transient exchange loses its bindings on a RabbitMQ restart while the durable queues survive, so the pair is kept consistent; both are metadata only and neither costs per-message I/O.
Both values remain configurable, so an installation that needs the old behaviour can still set DurableQueues = false - on a RabbitMQ old enough to accept it.
Tested with Naemon 1.4.1 against RabbitMQ 3.9.27, publishing to all configured queues, alongside the Go worker consuming them. Because both sides declare the same queues and AMQP answers a mismatched redeclare with a 406 PRECONDITION_FAILED rather than reconciling it, this change belongs with the matching one in Statusengine Go Worker; both start orders were checked, module first and worker first, with no 406 either way.
Note for existing installations: the queues already exist as non-durable and cannot be redeclared. They have to be deleted once, with the monitoring core and the worker stopped. Messages waiting in them are lost, which is acceptable for the same reason the design is - they are transient and would not have survived a broker restart either.