haproxy unable to start; None referenced in haproxy.conf

Sosreports for the octavia/2 unit have been uploaded and reference SF ticket #246626.

Hello,
Is it possible to have an update on this issue,please? Our customer requires a status update.
Thank you.

I'm fairly sure this is related to bug 1858132

Fixes related to the bug referenced above where included in the 20.05 charm release so its quite possible this issue has been resolved.

Ryan - I looked at SF #246626 but it seemed unrelated to this bug report - please either provide the correct reference or attach the crashdump directly to this bug.

backend octavia-api_admin_10.246.65.140
  balance leastconn
  server octavia-2 10.246.65.140:9866 check
  server octavia-0 10.246.65.169:9866 check
  server octavia-1 None:9866 check
  server octavia-3 10.246.65.110:9866 check

looks odd - it would appear that octavia/1 is still in the related unit list, but is not presenting any data about addresses.

Why that occurs - I don't know - but we can make the peer list build out more defensive:

https://review.opendev.org/741882

There was a similar bug for K8s charms:

https://github.com/johnsca/juju-relation-mysql/issues/5

Looking at the Juju code, entries in mongodb for relation settings are created earlier than the scope doc so the charms only know about relations for which there are already relation settings present.

https://github.com/juju/juju/blob/juju-2.7.8/apiserver/facades/agent/uniter/uniter.go#L1461-L1483 (ingress-address and private-address get placed into relation settings)
https://github.com/juju/juju/blob/juju-2.7.8/state/relationunit.go#L112-L150 (transaction ordering: settings go first then goes the scope doc)

However, looking more closely, inability to retrieve address information is not an error condition (merely a warning):

https://github.com/juju/juju/blob/juju-2.7.8/apiserver/facades/agent/uniter/uniter.go#L1476-L1479

There seem to be conditions in NetworksForRelation where an address may not be available:

https://github.com/juju/juju/blob/juju-2.7.8/apiserver/facades/agent/uniter/networkinfo.go#L281-L308

And these two Juju bugs are related to this:

https://bugs.launchpad.net/juju/+bug/1830252
https://bugs.launchpad.net/juju/+bug/1848628

I am going to add Juju to comment - if ingress-address is not guaranteed to be present in relation settings during the handling of <relname>-relation-joined then it would be best to document it for both IAAS and CAAS models. I couldn't find any reference to this behavior in the current docs.

Fix proposed to branch: master
Review: https://review.opendev.org/742413

Change abandoned by James Page (<email address hidden>) on branch: master
Review: https://review.opendev.org/742413
Reason: Rebuild ahead of this one picked up the interface fixes.

Another review landed ahead of mine picking up the required interface update in openstack-ha.

If I read it correctly for IAAS models, we should always have an ingress-address at the point where a unit is running hooks.

The machine agent sets (machine) addresses when it starts, so before units land on a machine it will have a value for its preferred public and private addresses.

Unit addresses (requested in NetworksForRelation mentioned above) are derived from the machine they are running on.

Marked as invalid for Juju, because I don't think that we have concrete evidence that the bug is in Juju, and the charm has worked around it. Feel free to post more evidence and reopen if necessary :-)

As manadart observed a few days, ago, for IAAS models, the ingress-address should always be present. Given that we are seeing IAAS models where ingress-address is not set, it seems to be an acknowledged bug in Juju, rather than invalid.

Re-opening as I have been convinced that this might be a Juju issue after all.

The next steps here are to try to reproduce it so that we can prove that it is a Juju bug and fix it.

According to this:
octavia/2 /etc/haproxy/haproxy.cfg:

backend octavia-api_admin_10.246.65.140
  balance leastconn
  server octavia-2 10.246.65.140:9866 check
  server octavia-0 10.246.65.169:9866 check
  server octavia-1 None:9866 check
  server octavia-3 10.246.65.110:9866 check

vs
octavia/0 - just right
backend octavia-api_admin_10.246.65.169
    balance leastconn
    server octavia-0 10.246.65.169:9866 check
    server octavia-2 10.246.65.140:9866 check
    server octavia-3 10.246.65.110:9866 check

And the original statement:
A server was taken down for maintenance. It's old units were removed from the juju model. Then re-added when the host was back in commission.

It sounds like octavia-1 was removed, and then when the machine was back up, we introduced octavia-3.

Which means the fact that octavia-2 is still thinking about octavia-1 is the problem, as octavia-1 definitely doesn't have an IP address anymore.

Now I don't know why octavia-3 doesn't know about octavia-2
octavia/3 - too few
backend octavia-api_admin_10.246.65.110
    balance leastconn
    server octavia-3 10.246.65.110:9866 check
    server octavia-0 10.246.65.169:9866 check

Potentially the fact that octavia-2 is still thinking about octavia-1 is causing a problem in the charm, causing it to not respond to relation-joined from octavia-3, so octavia-3 is not then seeing octavia-2. If that were happening, I'd definitely expect to see charms in error state and errors in the Juju unit logs.

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.