nova dns lookups can block the nova api process leading to 503 errors.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
sean mooney | ||
Train |
In Progress
|
Undecided
|
Unassigned | ||
Ussuri |
In Progress
|
Undecided
|
Unassigned | ||
Victoria |
In Progress
|
Undecided
|
Unassigned | ||
Wallaby |
Fix Released
|
Undecided
|
Unassigned | ||
Xena |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
we currently have 4 possibly related downstream bugs whereby DNS lookups can
result in 503 errors as we do not monkey patch green DNS and that can result in blocking behavior.
specifically we have seen callses to socket.getaddrinfo in py-amqp block the API
when using ipv6.
https:/
https:/
https:/
https:/
copying a summary of the rca
from one of the bugs
What happens:
- A request comes in which requires rpc, so a new connection to rabbitmq is to be established
- The hostname(s) from the transport_url setting are ultimately passed to py-amqp, which attempts to resolve the hostname to an ip address so it can set up the underlying socket and connect
- py-amqp explicitly tries to resolve with AF_INET first and then only if that fails, then it tries with AF_INET6[1]
- The customer environment is primarily IPv6. Attempting to resolve the hostname via AF_INET fails nss_hosts (the /etc/hosts file only have IPv6 addrs), and falls through to nss_dns
- Something about the customer DNS infrastructure is slow, so it takes a long time (~10 seconds) for this IPv4-lookup to fail.
- py-amqp finally tries with AF_INET6 and the hostname is resolved immediately via nss_hosts because the entry is in the /etc/hosts
Critically, because nova explicitly disables greendns[2] with eventlet, the *entire* nova-api worker is blocked during the duration of the slow name resolution, because socket.getaddrinfo is a blocking call into glibc.
[1] https:/
[2] https:/
nova currently disables greendns monkeypatch because of a very old bug on centos 6 on python 2.6 and the havana release of nova https:/
ipv6 support was added in v0.17 in the same release that added python 3 support back in 2015
https:/
so we should not need to work around the lack of ipv6 support anymore.
https:/
Changed in nova: | |
status: | Triaged → In Progress |
nit: technially this could block any dns lookup not just in the API such as in the compute agent when making calls to neuton exctra.