| 2021-09-29 17:40:51 |
Simon Déziel |
description |
The problem is twofold:
1) randomly calling relation_departed:
When adding an additional unit, the existing unit had it's relation_departed hook triggered out of the blue, apparently when I added an extra unit. Here's what I've done:
# deploy initial unit (https-client/33):
$ juju deploy ./https-client_ubuntu-20.04-amd64.charm
Located local charm "https-client", revision 30
Deploying "https-client" from local charm "https-client", revision 30
# establish relation with lxd:
$ juju relate https-client lxd
# so far so go so let's add one more unit (https-client/34):
$ juju add-unit https-client
# check the juju logs:
unit-https-client-34: 16:31:28 INFO juju Starting unit workers for "https-client/34"
unit-https-client-34: 16:31:28 INFO juju.worker.apicaller [5c4e7f] "unit-https-client-34" successfully connected to "[2602:fc62:b:1018:1::a025]:17070"
unit-https-client-34: 16:31:28 INFO juju.worker.apicaller [5c4e7f] password changed for "unit-https-client-34"
unit-https-client-34: 16:31:28 INFO juju.worker.apicaller [5c4e7f] "unit-https-client-34" successfully connected to "172.17.18.217:17070"
unit-https-client-34: 16:31:28 INFO juju.worker.migrationminion migration phase is now: NONE
unit-https-client-34: 16:31:28 INFO juju.worker.logger logger worker started
unit-https-client-34: 16:31:28 INFO juju.worker.upgrader no waiter, upgrader is done
unit-https-client-33: 16:31:29 DEBUG unit.https-client/33.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-https-client-33: 16:31:29 DEBUG unit.https-client/33.juju-log https:39: Legacy hooks/https-relation-departed does not exist.
unit-https-client-33: 16:31:29 DEBUG unit.https-client/33.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-https-client-33/charm/.unit-state.db already exists
unit-https-client-33: 16:31:29 DEBUG unit.https-client/33.juju-log https:39: Emitting Juju event https_relation_departed.
unit-lxd-21: 16:31:29 DEBUG unit.lxd/21.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-lxd-21: 16:31:29 DEBUG unit.lxd/21.juju-log https:39: Legacy hooks/https-relation-departed does not exist.
unit-lxd-21: 16:31:29 DEBUG unit.lxd/21.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-lxd-21/charm/.unit-state.db already exists
unit-lxd-21: 16:31:29 DEBUG unit.lxd/21.juju-log https:39: Emitting Juju event https_relation_departed.
unit-lxd-21: 16:31:29 DEBUG unit.lxd/21.juju-log https:39: _on_https_relation_departed
unit-lxd-21: 16:31:29 INFO unit.lxd/21.juju-log https:39: Deleting juju-relation-test-https-client/33:autoremove from trusted list
In the above, we see https-client/33's departed hook was called for no reason.
2) not calling relation_departed:
# after the unexpected departure of https-client/33, I added another unit (https-client/35):
$ juju add-unit https-client
# this one behaved fine (joined and didn't cause any disruption)
# now to break the relation:
$ juju remove-relation https-client lxd
# check the juju logs:
unit-lxd-21: 16:35:36 DEBUG unit.lxd/21.juju-log Operator Framework 1.2.0 up and running.
unit-lxd-21: 16:35:36 DEBUG unit.lxd/21.juju-log Legacy hooks/update-status does not exist.
unit-lxd-21: 16:35:36 DEBUG unit.lxd/21.juju-log Using local storage: /var/lib/juju/agents/unit-lxd-21/charm/.unit-state.db already exists
unit-lxd-21: 16:35:36 DEBUG unit.lxd/21.juju-log Emitting Juju event update_status.
unit-https-client-34: 16:35:47 DEBUG unit.https-client/34.juju-log Operator Framework 1.2.0 up and running.
unit-https-client-34: 16:35:47 DEBUG unit.https-client/34.juju-log Legacy hooks/update-status does not exist.
unit-https-client-34: 16:35:47 DEBUG unit.https-client/34.juju-log Using local storage: /var/lib/juju/agents/unit-https-client-34/charm/.unit-state.db already exists
unit-https-client-34: 16:35:47 DEBUG unit.https-client/34.juju-log Emitting Juju event update_status.
unit-https-client-35: 16:35:50 DEBUG unit.https-client/35.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-https-client-34: 16:35:50 DEBUG unit.https-client/34.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-https-client-34: 16:35:50 DEBUG unit.https-client/34.juju-log https:39: Legacy hooks/https-relation-broken does not exist.
unit-https-client-35: 16:35:50 DEBUG unit.https-client/35.juju-log https:39: Legacy hooks/https-relation-broken does not exist.
unit-https-client-33: 16:35:50 DEBUG unit.https-client/33.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-https-client-33: 16:35:50 DEBUG unit.https-client/33.juju-log https:39: Legacy hooks/https-relation-broken does not exist.
unit-https-client-35: 16:35:50 DEBUG unit.https-client/35.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-https-client-35/charm/.unit-state.db already exists
unit-https-client-33: 16:35:50 DEBUG unit.https-client/33.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-https-client-33/charm/.unit-state.db already exists
unit-https-client-34: 16:35:50 DEBUG unit.https-client/34.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-https-client-34/charm/.unit-state.db already exists
unit-https-client-35: 16:35:50 DEBUG unit.https-client/35.juju-log https:39: Emitting Juju event https_relation_broken.
unit-https-client-33: 16:35:50 DEBUG unit.https-client/33.juju-log https:39: Emitting Juju event https_relation_broken.
unit-https-client-34: 16:35:50 DEBUG unit.https-client/34.juju-log https:39: Emitting Juju event https_relation_broken.
unit-lxd-21: 16:35:51 DEBUG unit.lxd/21.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-lxd-21: 16:35:51 DEBUG unit.lxd/21.juju-log https:39: Legacy hooks/https-relation-departed does not exist.
unit-lxd-21: 16:35:51 DEBUG unit.lxd/21.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-lxd-21/charm/.unit-state.db already exists
unit-lxd-21: 16:35:51 DEBUG unit.lxd/21.juju-log https:39: Emitting Juju event https_relation_departed.
unit-lxd-21: 16:35:51 DEBUG unit.lxd/21.juju-log https:39: _on_https_relation_departed
unit-lxd-21: 16:35:51 INFO unit.lxd/21.juju-log https:39: Deleting juju-relation-test-https-client/34:autoremove from trusted list
In the above, none of the https-client units saw their relation_departed hook invoked, only their (non-existent) relation_broken hook. For some reason, the lxd unit received a relation_departed for https-client/34 (but no other).
Additional information:
$ juju --version
2.9.15-ubuntu-amd64
$ juju status --relations --color
Model Controller Cloud/Region Version SLA Timestamp
test overlord maas/default 2.9.14 unsupported 16:44:05Z
App Version Status Scale Charm Store Channel Rev OS Message
https-client active 3 https-client local 30 ubuntu
lxd active 1 lxd local 30 ubuntu
Unit Workload Agent Machine Public address Ports Message
https-client/33* active idle 68 2602:fc62:b:1018:0:1:0:c
https-client/34 active idle 69 2602:fc62:b:1018:0:1:0:f
https-client/35 active idle 72 2602:fc62:b:1018:0:1:0:e
lxd/21* active idle 57 2602:fc62:b:1018:0:1:0:b
Machine State DNS Inst id Series AZ Message
57 started 2602:fc62:b:1018:0:1:0:b cloud-vm08 focal default Deployed
68 started 2602:fc62:b:1018:0:1:0:c cloud-vm03 focal default Deployed
69 started 2602:fc62:b:1018:0:1:0:f cloud-vm04 focal default Deployed
71 started 2602:fc62:b:1018:0:1:0:7 cloud-vm05 focal default Deployed
72 started 2602:fc62:b:1018:0:1:0:e cloud-vm06 focal default Deployed
73 started 2602:fc62:b:1018:0:1:0:2 cloud-vm07 focal default Deployed
74 started 2602:fc62:b:1018:0:1:0:3 cloud-vm10 focal default Deployed
75 started 2602:fc62:b:1018:0:1:0:d cloud-vm11 focal default Deployed
76 started 2602:fc62:b:1018:0:1:0:12 cloud-vm14 focal default Deployed
77 started 2602:fc62:b:1018:0:1:: cloud-vm01 focal default Deployed
78 started 2602:fc62:b:1018:0:1:0:13 cloud-vm12 focal default Deployed
79 started 2602:fc62:b:1018:0:1:0:14 cloud-vm13 focal default Deployed
81 started 2602:fc62:b:1018:0:1:0:5 cloud-vm09 focal default Deployed
82 pending pending focal |
The problem is twofold:
1) randomly calling relation_departed:
When adding an additional unit, the existing unit had it's relation_departed hook triggered out of the blue, apparently when I added an extra unit. Here's what I've done:
# deploy initial unit (https-client/33):
$ juju deploy ./https-client_ubuntu-20.04-amd64.charm
Located local charm "https-client", revision 30
Deploying "https-client" from local charm "https-client", revision 30
# establish relation with lxd:
$ juju relate https-client lxd
# so far so good so let's add one more unit (https-client/34):
$ juju add-unit https-client
# check the juju logs:
unit-https-client-34: 16:31:28 INFO juju Starting unit workers for "https-client/34"
unit-https-client-34: 16:31:28 INFO juju.worker.apicaller [5c4e7f] "unit-https-client-34" successfully connected to "[2602:fc62:b:1018:1::a025]:17070"
unit-https-client-34: 16:31:28 INFO juju.worker.apicaller [5c4e7f] password changed for "unit-https-client-34"
unit-https-client-34: 16:31:28 INFO juju.worker.apicaller [5c4e7f] "unit-https-client-34" successfully connected to "172.17.18.217:17070"
unit-https-client-34: 16:31:28 INFO juju.worker.migrationminion migration phase is now: NONE
unit-https-client-34: 16:31:28 INFO juju.worker.logger logger worker started
unit-https-client-34: 16:31:28 INFO juju.worker.upgrader no waiter, upgrader is done
unit-https-client-33: 16:31:29 DEBUG unit.https-client/33.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-https-client-33: 16:31:29 DEBUG unit.https-client/33.juju-log https:39: Legacy hooks/https-relation-departed does not exist.
unit-https-client-33: 16:31:29 DEBUG unit.https-client/33.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-https-client-33/charm/.unit-state.db already exists
unit-https-client-33: 16:31:29 DEBUG unit.https-client/33.juju-log https:39: Emitting Juju event https_relation_departed.
unit-lxd-21: 16:31:29 DEBUG unit.lxd/21.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-lxd-21: 16:31:29 DEBUG unit.lxd/21.juju-log https:39: Legacy hooks/https-relation-departed does not exist.
unit-lxd-21: 16:31:29 DEBUG unit.lxd/21.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-lxd-21/charm/.unit-state.db already exists
unit-lxd-21: 16:31:29 DEBUG unit.lxd/21.juju-log https:39: Emitting Juju event https_relation_departed.
unit-lxd-21: 16:31:29 DEBUG unit.lxd/21.juju-log https:39: _on_https_relation_departed
unit-lxd-21: 16:31:29 INFO unit.lxd/21.juju-log https:39: Deleting juju-relation-test-https-client/33:autoremove from trusted list
In the above, we see https-client/33's departed hook was called for no reason.
2) not calling relation_departed:
# after the unexpected departure of https-client/33, I added another unit (https-client/35):
$ juju add-unit https-client
# this one behaved fine (joined and didn't cause any disruption)
# now to break the relation:
$ juju remove-relation https-client lxd
# check the juju logs:
unit-lxd-21: 16:35:36 DEBUG unit.lxd/21.juju-log Operator Framework 1.2.0 up and running.
unit-lxd-21: 16:35:36 DEBUG unit.lxd/21.juju-log Legacy hooks/update-status does not exist.
unit-lxd-21: 16:35:36 DEBUG unit.lxd/21.juju-log Using local storage: /var/lib/juju/agents/unit-lxd-21/charm/.unit-state.db already exists
unit-lxd-21: 16:35:36 DEBUG unit.lxd/21.juju-log Emitting Juju event update_status.
unit-https-client-34: 16:35:47 DEBUG unit.https-client/34.juju-log Operator Framework 1.2.0 up and running.
unit-https-client-34: 16:35:47 DEBUG unit.https-client/34.juju-log Legacy hooks/update-status does not exist.
unit-https-client-34: 16:35:47 DEBUG unit.https-client/34.juju-log Using local storage: /var/lib/juju/agents/unit-https-client-34/charm/.unit-state.db already exists
unit-https-client-34: 16:35:47 DEBUG unit.https-client/34.juju-log Emitting Juju event update_status.
unit-https-client-35: 16:35:50 DEBUG unit.https-client/35.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-https-client-34: 16:35:50 DEBUG unit.https-client/34.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-https-client-34: 16:35:50 DEBUG unit.https-client/34.juju-log https:39: Legacy hooks/https-relation-broken does not exist.
unit-https-client-35: 16:35:50 DEBUG unit.https-client/35.juju-log https:39: Legacy hooks/https-relation-broken does not exist.
unit-https-client-33: 16:35:50 DEBUG unit.https-client/33.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-https-client-33: 16:35:50 DEBUG unit.https-client/33.juju-log https:39: Legacy hooks/https-relation-broken does not exist.
unit-https-client-35: 16:35:50 DEBUG unit.https-client/35.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-https-client-35/charm/.unit-state.db already exists
unit-https-client-33: 16:35:50 DEBUG unit.https-client/33.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-https-client-33/charm/.unit-state.db already exists
unit-https-client-34: 16:35:50 DEBUG unit.https-client/34.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-https-client-34/charm/.unit-state.db already exists
unit-https-client-35: 16:35:50 DEBUG unit.https-client/35.juju-log https:39: Emitting Juju event https_relation_broken.
unit-https-client-33: 16:35:50 DEBUG unit.https-client/33.juju-log https:39: Emitting Juju event https_relation_broken.
unit-https-client-34: 16:35:50 DEBUG unit.https-client/34.juju-log https:39: Emitting Juju event https_relation_broken.
unit-lxd-21: 16:35:51 DEBUG unit.lxd/21.juju-log https:39: Operator Framework 1.2.0 up and running.
unit-lxd-21: 16:35:51 DEBUG unit.lxd/21.juju-log https:39: Legacy hooks/https-relation-departed does not exist.
unit-lxd-21: 16:35:51 DEBUG unit.lxd/21.juju-log https:39: Using local storage: /var/lib/juju/agents/unit-lxd-21/charm/.unit-state.db already exists
unit-lxd-21: 16:35:51 DEBUG unit.lxd/21.juju-log https:39: Emitting Juju event https_relation_departed.
unit-lxd-21: 16:35:51 DEBUG unit.lxd/21.juju-log https:39: _on_https_relation_departed
unit-lxd-21: 16:35:51 INFO unit.lxd/21.juju-log https:39: Deleting juju-relation-test-https-client/34:autoremove from trusted list
In the above, none of the https-client units saw their relation_departed hook invoked, only their (non-existent) relation_broken hook. For some reason, the lxd unit received a relation_departed for https-client/34 (but no other).
Additional information:
$ juju --version
2.9.15-ubuntu-amd64
$ juju status --relations --color
Model Controller Cloud/Region Version SLA Timestamp
test overlord maas/default 2.9.14 unsupported 16:44:05Z
App Version Status Scale Charm Store Channel Rev OS Message
https-client active 3 https-client local 30 ubuntu
lxd active 1 lxd local 30 ubuntu
Unit Workload Agent Machine Public address Ports Message
https-client/33* active idle 68 2602:fc62:b:1018:0:1:0:c
https-client/34 active idle 69 2602:fc62:b:1018:0:1:0:f
https-client/35 active idle 72 2602:fc62:b:1018:0:1:0:e
lxd/21* active idle 57 2602:fc62:b:1018:0:1:0:b
Machine State DNS Inst id Series AZ Message
57 started 2602:fc62:b:1018:0:1:0:b cloud-vm08 focal default Deployed
68 started 2602:fc62:b:1018:0:1:0:c cloud-vm03 focal default Deployed
69 started 2602:fc62:b:1018:0:1:0:f cloud-vm04 focal default Deployed
71 started 2602:fc62:b:1018:0:1:0:7 cloud-vm05 focal default Deployed
72 started 2602:fc62:b:1018:0:1:0:e cloud-vm06 focal default Deployed
73 started 2602:fc62:b:1018:0:1:0:2 cloud-vm07 focal default Deployed
74 started 2602:fc62:b:1018:0:1:0:3 cloud-vm10 focal default Deployed
75 started 2602:fc62:b:1018:0:1:0:d cloud-vm11 focal default Deployed
76 started 2602:fc62:b:1018:0:1:0:12 cloud-vm14 focal default Deployed
77 started 2602:fc62:b:1018:0:1:: cloud-vm01 focal default Deployed
78 started 2602:fc62:b:1018:0:1:0:13 cloud-vm12 focal default Deployed
79 started 2602:fc62:b:1018:0:1:0:14 cloud-vm13 focal default Deployed
81 started 2602:fc62:b:1018:0:1:0:5 cloud-vm09 focal default Deployed
82 pending pending focal |
|
