leader_id stale/incorrect; causes rsync cron job missing on leader unit
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Ubuntu Repository Cache Charm |
Fix Released
|
High
|
Haw Loeung | ||
Bug Description
Hi,
As seen today, leader_id was stale/incorrect:
| Unit Workload Agent Machine Public address Ports Message
| ubuntu-
| ...
| ubuntu-
| ...
| ubuntu@machine-0:~$ sudo juju-run ubuntu-
| leader_id: ubuntu-
| ubuntu@machine-2:~$ sudo juju-run ubuntu-
| leader_id: ubuntu-
leader_id only gets set on leader-elected hook firing. I think we should also have it run on config-changed or some other to ensure that leader_id isn't stale.
Bit of evidence - https:/
| 2018-10-11 01:09:20 WARNING juju-log cluster:1: Leader changed between peer_update_
Or even when the sync job runs from cron:
| 2018-10-11 02:23:36,164 - Executing hook: ['juju-run', 'ubuntu-
Have hooks/ubuntu-
Related branches
- Barry Price: Approve
- Canonical IS Reviewers: Pending requested
-
Diff: 22 lines (+12/-0)1 file modifiedreactive/ubuntu-repository-cache.py (+12/-0)
| description: | updated |
| Stuart Bishop (stub) wrote | #1 |
| Stuart Bishop (stub) wrote | #2 |
| Haw Loeung (hloeung) wrote | #3 |
| Changed in ubuntu-repository-cache: | |
| status: | New → Triaged |
| importance: | Undecided → High |
| Changed in ubuntu-repository-cache: | |
| assignee: | nobody → Haw Loeung (hloeung) |
| Changed in ubuntu-repository-cache: | |
| status: | Triaged → In Progress |
| Haw Loeung (hloeung) wrote | #4 |
| summary: |
- leader_id stale/incorrect + leader_id stale/incorrect; causes rsync cron job missing on leader unit |
| Haw Loeung (hloeung) wrote | #5 |
| Haw Loeung (hloeung) wrote | #6 |
| Changed in ubuntu-repository-cache: | |
| status: | In Progress → Fix Committed |
| Changed in ubuntu-repository-cache: | |
| status: | Fix Committed → Fix Released |
Updating this leader setting more aggressively will help, and a quick fix so can be done.
However, a setting like this is not reliable - it can only ever state which unit *was* the leader, and cannot state which unit *is* the leader. The only unit that can reliably know who the leader is is the leader itself (by calling is-leader). We should drop this leadership setting, and the charm refactored to only use 'is-leader' and not 'is-leader'. To communicate with the leader, place the message on the per relation. Any message being sent to a specific unit because it *was* the leader is a bug, because there is no guarantee that the unit will still be the leader when the message arrives.