Comment 0 for bug 1954748

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote : ovn-central crashed during update-status due to dns failure

The ovn-central charm is doing DNS lookups during the update-status hook and this crashed the charm. Although it's a reactive charm, it probably shouldn't be doing this much work, and should certainly handle a DNS lookup failure better than just crashing:

2021-12-13 16:42:36 ERROR juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-ovn-central-0/charm/reactive/layer_openstack.py", line 134, in default_request_certificates
    for cn, req in instance.get_certificate_requests().items():
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/classes.py", line 290, in get_certificate_requests
    return cert_utils.get_certificate_request(
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 142, in get_certificate_request
    req.add_hostname_cn()
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 93, in add_hostname_cn
    'cn': get_hostname(ip),
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/charmhelpers/contrib/network/ip.py", line 523, in get_hostname
    result = ns_query(rev)
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/charmhelpers/contrib/network/ip.py", line 479, in ns_query
    answers = dns.resolver.query(address, rtype)
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/dns/resolver.py", line 1100, in query
    return get_default_resolver().query(qname, rdtype, rdclass, tcp, source,
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.8/site-packages/dns/resolver.py", line 898, in query
    raise NoNameservers(request=request, errors=errors)
dns.resolver.NoNameservers: All nameservers failed to answer the query 158.0.16.172.in-addr.arpa. IN PTR: Server 127.0.0.53 UDP port 53 answered SERVFAIL

This happened on ServerStack which does have a known propensity for DNS lookup failures.