unhealthy juju-db is invisible

Bug #1806760 reported by Paul Collins
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
Low
Unassigned

Bug Description

I just discovered that the mongodb replicaset of one of our Juju controllers was unhealthy as I was preparing for an upgrade, because juju create-backup was attempted on a secondary (this is a bug in itself, LP:1790875, but it was helpful here) that was in RECOVERING state.

mongodb is a critical wad of dark matter underpinning Juju that we have to deal with directly in order to monitor and manage. Juju should at least have some way of making the operator aware of an unhealthy database so that they can start Googling feverishly for help on fixing it.

Revision history for this message
Richard Harding (rharding) wrote :

Thank you. This falls directly into the work we want to do around an easy to consume health report for Juju controllers, in particular around HA status and mongo status.

Changed in juju:
status: New → Triaged
importance: Undecided → Medium
milestone: none → 2.6-beta1
Changed in juju:
milestone: 2.6-beta1 → 2.6-beta2
Changed in juju:
milestone: 2.6-beta2 → 2.6-rc1
Changed in juju:
milestone: 2.6-rc1 → 2.6-rc2
Changed in juju:
milestone: 2.6-rc2 → 2.6.1
Changed in juju:
milestone: 2.6.1 → 2.6.2
Changed in juju:
milestone: 2.6.2 → 2.6.3
Changed in juju:
milestone: 2.6.3 → 2.6.4
Changed in juju:
milestone: 2.6.4 → 2.6.5
Changed in juju:
milestone: 2.6.5 → 2.6.6
Changed in juju:
milestone: 2.6.6 → 2.6.7
Changed in juju:
milestone: 2.6.7 → 2.7-beta1
Changed in juju:
milestone: 2.7-beta1 → 2.7-rc1
Changed in juju:
milestone: 2.7-rc1 → none
tags: added: health-check
Revision history for this message
Peter De Sousa (pjds) wrote :

hit this, the controller upgrade was failing stating one of the units was in state "RECOVERING".

To workaround this issue, check the logs of the units, finding the unit stating that Mongodb is too far behind to re-sync.

Then follow the steps on the mongodb docs to re-sync the DB.

https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/

Revision history for this message
Canonical Juju QA Bot (juju-qa-bot) wrote :

This Medium-priority bug has not been updated in 60 days, so we're marking it Low importance. If you believe this is incorrect, please update the importance.

Changed in juju:
importance: Medium → Low
tags: added: expirebugs-bot
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.