Purging rabbitmq queues

At some point we (or better, Nagios) observed that rabbitmq was queueing up messages and there was noo ne consuming them:

rabbitmqctl list_queues -p openstack | sort -k 2 -g

Eventually, the issue was nailed down to notifications.info and notifications_designate.info queues:

rabbitmqctl list_queues -p openstack name consumers messages | egrep "notifications(_designate)?\."

Specifically, notifications.info is supposed to be consumed by ceilometer, which was not installed at that time.

Solution

Queues cleaning involves both purging them and establishing a policy to automatically prune messages older than some timeout. Policies are shown with command:

rabbitmqctl list_policies -p openstack

Commands below need to be executed just once, on any of the RabbitMQ nodes in the cluster.

To purge a queue, execute something like:

rabbitmqctl purge_queue -p openstack notifications.info

and similarly for all other queues you need to purge.

For sake of simplicity, we set a reasonable timeout on any queue (timeout set to 20 minutes), and a more strict one on notifications* as well as alarm* queues (timeout set to 5 minutes), as both sets of queues are supposed to be consumed by ceilometer:

rabbitmqctl set_policy default-expiry -p openstack '^.*' '{"message-ttl":1200000,"ha-mode":"all","ha-sync-mode":"automatic"}' --apply-to queues --priority 10
rabbitmqctl set_policy alarm-expiry -p openstack '^alarm' '{"message-ttl":30000,"ha-mode":"all","ha-sync-mode":"automatic"}' --apply-to queues --priority 10
rabbitmqctl set_policy notifications-expiry -p openstack '^notifications' '{"message-ttl":30000,"ha-mode":"all","ha-sync-mode":"automatic"}' --apply-to queues --priority 10

In the end, our active policies now read:

root@juju-08eaf8-91-lxd-34:/home/ubuntu# rabbitmqctl list_policies -p openstack
Listing policies ...
openstack       HA      all     ^(?!amq\\.).*   {"ha-mode":"all","ha-sync-mode":"automatic"}    0
openstack       alarm-expiry    queues  ^alarm  {"message-ttl":30000,"ha-mode":"all","ha-sync-mode":"automatic"}        10
openstack       default-expiry  queues  ^.*     {"message-ttl":1200000,"ha-mode":"all","ha-sync-mode":"automatic"}      10
openstack       notifications-expiry    queues  ^notifications  {"message-ttl":30000,"ha-mode":"all","ha-sync-mode":"automatic"}        10