rabbitmq fails to start on servers with more than 42 cpus
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| OpenStack RabbitMQ Server Charm |
Fix Committed
|
Medium
|
Unassigned | ||
| Focal |
Fix Released
|
Undecided
|
Unassigned | ||
| Jammy |
Fix Released
|
Undecided
|
Unassigned | ||
Bug Description
Normally we run rabbitmq-server in a container but I was testing rabbitmq on a few bare-metal servers and all of them were coming up in 'error' state:
Unit Workload Agent Machine Public address Ports Message
rabbitmq-
rabbitmq-server/1 waiting executing 1 10.110.244.142 5672/tcp Waiting for all 3 peers to complete the cluster.
filebeat/1 waiting idle 10.110.244.142 Waiting for:
elasticsearch, logstash or kafka.
nrpe/1 active idle 10.110.244.142 icmp,5666/tcp ready
rabbitmq-server/2 error idle 2 10.110.244.143 5672/tcp hook failed: "cluster-
filebeat/3 waiting idle 10.110.244.143 Waiting for:
elasticsearch, logstash or kafka.
nrpe/3 active idle 10.110.244.143 icmp,5666/tcp ready
After connecting to one of the units and trying to start rabbit manually, I see this error:
# /usr/sbin/
bad number of async threads 1920
Usage: beam.smp [flags] [ -- [init_args] ]
The flags are:
-a size suggested stack size in kilo words for threads
in the async-thread pool, valid range is [16-8192]
-A number set number of threads in async thread pool,
[...]
The usage above state 1024 is the max. And a quick search reveals where that 1920 is getting set:
# cat /etc/rabbitmq/
#####
# [ WARNING ]
# Configuration file maintained by Juju. Local changes may be overwritten.
#####
RABBITMQ_
Turns out the charm sets that number based on a default multiplier (24) times the number of cpus on the server. In this case:
# grep processor /proc/cpuinfo | wc -l
80
# echo $((80*24))
1920
So, although this seems a rare thing, the charm code should impose a maximum value of 1024 for the size of the async thread pool.
| Changed in fuel: | |
| assignee: | nobody → Vern Hart (vhart) |
| affects: | fuel → charm-rabbitmq-server |
| Changed in charm-rabbitmq-server: | |
| status: | New → In Progress |
| Frode Nordahl (fnordahl) wrote | #1 |
| Changed in charm-rabbitmq-server: | |
| importance: | Undecided → Medium |
| OpenStack Infra (hudson-openstack) wrote Change abandoned on charm-rabbitmq-server (master) | #2 |
| Changed in charm-rabbitmq-server: | |
| status: | In Progress → Invalid |
| assignee: | Vern Hart (vern) → nobody |
| Changed in charm-rabbitmq-server: | |
| status: | Invalid → Triaged |
| Changed in charm-rabbitmq-server: | |
| status: | Triaged → In Progress |
| OpenStack Infra (hudson-openstack) wrote Fix merged to charm-rabbitmq-server (master) | #4 |
| Changed in charm-rabbitmq-server: | |
| status: | In Progress → Fix Committed |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to charm-rabbitmq-server (stable/jammy) | #5 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to charm-rabbitmq-server (stable/focal) | #6 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to charm-rabbitmq-server (stable/jammy) | #7 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to charm-rabbitmq-server (stable/focal) | #8 |
It seems OpenStack Infra did not automatically post the review link, so for reference; https:/