[2.9.31 testing] many charms lost connectivity to Juju

This looks like an api rate limiting situation, where the controller was rejecting too many concurrent api requests. What looks to be interesting is that the code which opens the api connection does not appear to account for the "try again" error and instead of backing off and retrying transparently, errors out. Now it should eventually come good - the error will cause the agent to restart but doing it this way is noisy, as opposed to silently retrying because the error is recognised as "try again".

That generally indicates something like the controller itself restarting in the middle of operations, causing all agents to reconnect and some of them getting rate limited. (Default settings is to allow 4 new connections per second). It might be interesting to go back a bit and see if there was something obvious that caused the controller to restart.

This Medium-priority bug has not been updated in 60 days, so we're marking it Low importance. If you believe this is incorrect, please update the importance.

Seen again on: https://solutions.qa.canonical.com/v2/testruns/52fdaaf2-a977-40ff-855a-2f47b907efa7/
Artifacts: https://oil-jenkins.canonical.com/artifacts/52fdaaf2-a977-40ff-855a-2f47b907efa7/index.html

Future occurrences: https://solutions.qa.canonical.com/v2/bugs/1976289