Check that the hostname resolves and has not changed. RabbitMQ names its node rabbit@hostname and stores its data under that name. A hostname that does not resolve, or one that changed since install, is the most RabbitMQ-specific cause of this failure.
hostname -s getent hosts $(hostname -s) # the log is named after the node sudo tail -60 /var/log/rabbitmq/rabbit@$(hostname -s).log
If getent prints nothing, add the short hostname to /etc/hosts against 127.0.1.1 and start the service again.
Why a renamed host breaks RabbitMQ
The data directory is /var/lib/rabbitmq/mnesia/rabbit@oldhost. Rename the machine, or move it to a cloud image that gets a new hostname at boot, and RabbitMQ looks for rabbit@newhost instead. Depending on version it either fails to start or starts with an empty database, which looks like every queue and user vanished. Neither is data loss: the old directory is still there. Pin the node name back with NODENAME=rabbit@oldhost in /etc/rabbitmq/rabbitmq-env.conf, or set the hostname back.
The other causes, in order
- Erlang cookie permissions.
/var/lib/rabbitmq/.erlang.cookiemust be owned by rabbitmq and readable by nobody else. Erlang refuses to start with a world readable cookie:sudo chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie sudo chmod 400 /var/lib/rabbitmq/.erlang.cookie
- epmd cannot be reached. The Erlang port mapper on 4369 has to be available before the node starts. A firewall rule, a container without loopback networking, or a stale epmd from an old Erlang install stops the node. The log mentions epmd or a distribution failure.
- A corrupt quorum queue log. After a crash or a full disk, a write-ahead log file under
mnesia/rabbit@host/quorum/can be truncated and stop the node on boot. The log names the .wal file. Moving that one file aside is the fix RabbitMQ maintainers give, but it can lose recent messages on quorum queues, so copy the whole mnesia directory first. - Erlang and RabbitMQ versions do not match. Each RabbitMQ release supports a narrow range of Erlang versions. Upgrading one through the distribution and the other through a vendor repository is the usual way this happens. Check both with
erl -versionanddpkg -l | grep rabbitmqagainst the RabbitMQ compatibility table. - Ports already bound. 5672 for AMQP, 25672 for clustering, 4369 for epmd:
sudo ss -tlnp | grep -E ':(5672|25672|4369) '. - Running inside an LXC or LXD container. The packaged unit asks for device and resource controls that unprivileged containers refuse, producing
Operation not permittedin the journal. Fix it on the container side, or override the unit’s restricting directives in a drop-in.
Reading the exit code
status=1/FAILURE means the node started and stopped, so read the node log above rather than the journal, which usually only says the process exited. An activating (auto-restart) state that never settles means systemd keeps retrying a node that keeps dying, and the cause is the same. The full exit-code table is in the nginx version of this error.
How to stop it happening again
sudo rabbitmq-diagnostics status sudo rabbitmq-diagnostics check_running
Pin NODENAME explicitly in rabbitmq-env.conf on any server whose hostname might change, which covers most cloud instances. That single line prevents the most confusing version of this failure. Keep Erlang and RabbitMQ from the same repository so one is never upgraded without the other.
Related
- Job for redis-server.service failed: another broker where the systemd unit Type is the usual cause.
- Job for elasticsearch.service failed: another clustered service that fails on host-level settings.
- Job for nginx.service failed: the same systemd error, plus the full exit-code table.
RabbitMQ started but every queue and user is gone
Almost certainly a hostname change. RabbitMQ started a fresh node under the new name and ignored the old data directory. List /var/lib/rabbitmq/mnesia/: a directory named after the old host is your data. Stop the service, set NODENAME back to the old name, and start it again.
The node log is empty
Then Erlang failed before RabbitMQ could open its log, which points at the cookie permissions or epmd rather than at RabbitMQ configuration. Check the journal with sudo journalctl -xeu rabbitmq-server --no-pager | tail -40 for the Erlang error.
Leave a Reply