Sending amphora heartbeats fails when the LB reaches a certain size
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
octavia |
New
|
Undecided
|
Unassigned |
Bug Description
The amphora-agent seems not to be able to send heartbeat messages larger than the max size of a UDP datagram.
We noticed this on a very large LB with 106 listeners, 106 pools and 26 members per pool.
Thats 2862 resources in total.
After adding some debugging output in octavia.
In that case the payload size of a UDP heartbeat was 66250 bytes.
It seems that we are limited to a length of 65507 bytes with UDP and IPv4.
As a result of that the octavia health manager is constantly triggering failovers on the amphoras of the LB.
On a new amphora, the first few heartbeats succeed until the haproxy config is being deployed.
After that no heartbeats go through.
This was observed on amphoras with Ubuntu 20.04.4 LTS and amphora-agent 9.0.2.dev34.
Thanks for reporting this issue.
We had a RFE on storyboard that mitigates this packet size issue: /storyboard. openstack. org/#!/ story/2010216
[RFE] Optionally send amphora heart beat messages via TCP https:/
A commit was proposed: /review. opendev. org/c/openstack /octavia/ +/852269
852269: Health sender: support to send via TCP | https:/
it got one CR+2 and needs more reviews