Canonical Juju

HA Juju controllers showing inconsistent status

Bug #2020849 reported by James Simpson
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
Undecided
Joseph Phillips

Bug Description

Hi,

We've experienced an issue with the consistency and stability of our Juju controllers, and are struggling to pinpoint what's actually happening.

We're operating a HA controller set, running Juju 2.9.42, deployed in an Openstack cloud.

Symptoms we've observed have been:

* Issues with the stability of relationship hooks in deployed models (we have observed issues with relationships being created, updated, and departed)

* Controllers returning inconsistent "juju status" results
  When running "juju status --debug" to make sure we get one result from each controller, we have observed that at least one controller will consistently return a different result than the other(s).
  For example, this paste shows both secondary controllers reporting the primary controller as "agent-lost", while the primary disagrees: https://pastebin.ubuntu.com/p/VHrwFbm79Z/

Controller logs from the period in question have been made available via secure portal https://juju-controller-reports.admin.canonical.com/ps5-prodstack-is/

Model logs for the specific model in which we observed relationship hook issues are located in "special-request" under that directory.

Please advise if there are any additional logs we should supply, any metrics we can gather from the time, or anything else.

Thanks!

Tags: canonical-is
Tom Haddon (mthaddon)
tags: added: canonical-is
Revision history for this message
Ian Booth (wallyworld) wrote :

The controllers share agent connectivity info (aka presence) using pubsub. I don't think there's an explicit delivery guarantee for such messages.

Logging to turn on would be

juju.worker.pubsub=TRACE
juju.worker.presence=TRACE

Logs could also contain messages matching the format string

"%p programming error, e.ch=%v did not accept %v - missing Unwatch?\nwatch source:\n%s"

Extra relation debug can be obtained by setting

juju.worker.uniter.relation=TRACE

Vitaly Antonenko (anvial)
Changed in juju:
status: New → Incomplete
Revision history for this message
James Simpson (jsimpso) wrote :

Thanks Ian - I've configured those via "juju model-config -m controller logging-config=...", next time we experience an de-sync event do we just need to grab the same logs as we provided above?

Junien Fridrick (axino)
Changed in juju:
status: Incomplete → New
Revision history for this message
Joseph Phillips (manadart) wrote :

Looking through the initial logs supplied, I can see on machine 2 that we were spewing 404s around 25/5.
https://pastebin.canonical.com/p/3ZkgCDRpCt/

It almost looks as though the HTTP server worker was down, then managed to come up. This *might* explain inconsistencies.

Revision history for this message
Joseph Phillips (manadart) wrote :

Then there were a great deal of transaction issues:
https://pastebin.canonical.com/p/HD36KQxrWv/

Revision history for this message
Joseph Phillips (manadart) wrote :

Connectivity issues continue for some time with connection resets and broken pipes, then the lease issues that inevitably follow along with model leadership issues.

Joseph Phillips (manadart)
Changed in juju:
status: New → Triaged
assignee: nobody → Joseph Phillips (manadart)
Revision history for this message
Joseph Phillips (manadart) wrote :

Based on the latest logs and their corresponding engine reports, I'm not seeing an obvious cause for inconsistencies.

Can we get some detail on what the alerts indicate when they are raised?

Errors are mostly the usual firewaller worker ones for ipv6-icmp, which will be fixed by upgrading. This should be done in the short term, because it will reduce load and log spam.

I can see a sudden jump up in load on ubuntu/2 here:
https://grafana.admin.canonical.com/d/sR1-JkYmz/juju2-controllers-thumpers?orgId=1&var-controller=prodstack5-prodstack5-prodstack-is&var-host=All&var-node=All&from=1691466766995&to=1691556622995

There's nothing to explain this, and it is obviously affecting API response times.

Revision history for this message
Romain Couturat (romaincout) wrote :

around 2023-10-06T13:56:52+00:00 , running
alias jsft='juju status --format tabular'
4 times in a row gave me 3 different results (look at landscape-client scale) https://pastebin.canonical.com/p/QCGgK4kf24/

Revision history for this message
Joseph Phillips (manadart) wrote :

When you're seeing this, please capture the output with --debug supplied to the status command.

That will give us some idea of which controller we're actually connecting to for the result.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.