Bug #2011278 " <name>_relation_departed is sometimes called rand... : Bugs : Canonical Juju

Revision history for this message

Simon Déziel (sdeziel) wrote on 2023-03-11:

#1

juju debug-log --level DEBUG --tail --exclude-module juju Edit (40.5 KiB, text/plain)

Revision history for this message

Simon Déziel (sdeziel) wrote on 2023-03-11:

#2

juju debug-log -m controller --replay Edit (91.2 KiB, text/plain)

Revision history for this message

Simon Déziel (sdeziel) wrote on 2023-03-11:

#3

The charm itself Edit (5.8 MiB, application/octet-stream)

Revision history for this message

Simon Déziel (sdeziel) wrote on 2023-03-11:

#4

Snap resource Edit (32.0 MiB, application/octet-stream)

tags:

added: lxd-cloud

Joseph Phillips (manadart) on 2023-03-16

description:	updated
Changed in juju:
status:	New → Triaged
importance:	Undecided → High
assignee:	nobody → Joseph Phillips (manadart)

Revision history for this message

Simon Déziel (sdeziel) wrote on 2023-04-28 (last edit on 2023-04-28):

#5

Download full text (6.6 KiB)

I ran into this again but this time with juju 3.1.2:

$ juju deploy ./lxd-cloud-cell_ubuntu-22.04-amd64.charm cell-a --resource snap=./lxd-cloud-cell_0+git.6c40e0c_amd64.snap --config name=foo -n 3
Located local charm "lxd-cloud-cell", revision 34
Deploying "cell-a" from local charm "lxd-cloud-cell", revision 34 on ubuntu@22.04/stable

$ juju debug-log --color --level DEBUG --replay | grep -E 'cell-a/(2[89]|30)\.' | grep event
unit-cell-a-28: 20:31:59 DEBUG unit.cell-a/28.juju-log Emitting Juju event install.
unit-cell-a-29: 20:31:59 DEBUG unit.cell-a/29.juju-log Emitting Juju event install.
unit-cell-a-30: 20:31:59 DEBUG unit.cell-a/30.juju-log Emitting Juju event install.
unit-cell-a-29: 20:32:02 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_created.
unit-cell-a-28: 20:32:02 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_created.
unit-cell-a-30: 20:32:02 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_created.
unit-cell-a-29: 20:32:02 DEBUG unit.cell-a/29.juju-log Emitting Juju event leader_settings_changed.
unit-cell-a-28: 20:32:02 DEBUG unit.cell-a/28.juju-log Emitting Juju event leader_settings_changed.
unit-cell-a-30: 20:32:03 DEBUG unit.cell-a/30.juju-log Emitting Juju event leader_settings_changed.
unit-cell-a-29: 20:32:03 DEBUG unit.cell-a/29.juju-log Emitting Juju event config_changed.
unit-cell-a-28: 20:32:03 DEBUG unit.cell-a/28.juju-log Emitting Juju event config_changed.
unit-cell-a-30: 20:32:03 DEBUG unit.cell-a/30.juju-log Emitting Juju event config_changed.
unit-cell-a-29: 20:32:06 DEBUG unit.cell-a/29.juju-log Emitting Juju event start.
unit-cell-a-28: 20:32:06 DEBUG unit.cell-a/28.juju-log Emitting Juju event start.
unit-cell-a-29: 20:32:07 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-28: 20:32:07 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-30: 20:32:07 DEBUG unit.cell-a/30.juju-log Emitting Juju event start.
unit-cell-a-29: 20:32:07 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-28: 20:32:07 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-30: 20:32:07 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-30: 20:32:07 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-29: 20:32:08 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-28: 20:32:08 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-29: 20:32:08 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-28: 20:32:08 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-30: 20:32:08 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-30: 20:32:09 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell...

I ran into this again but this time with juju 3.1.2:

$ juju deploy ./lxd-cloud-cell_ubuntu-22.04-amd64.charm cell-a --resource snap=./lxd-cloud-cell_0+git.6c40e0c_amd64.snap --config name=foo -n 3
Located local charm "lxd-cloud-cell", revision 34
Deploying "cell-a" from local charm "lxd-cloud-cell", revision 34 on ubuntu@22.04/stable

$ juju debug-log --color --level DEBUG --replay | grep -E 'cell-a/(2[89]|30)\.' | grep event
unit-cell-a-28: 20:31:59 DEBUG unit.cell-a/28.juju-log Emitting Juju event install.
unit-cell-a-29: 20:31:59 DEBUG unit.cell-a/29.juju-log Emitting Juju event install.
unit-cell-a-30: 20:31:59 DEBUG unit.cell-a/30.juju-log Emitting Juju event install.
unit-cell-a-29: 20:32:02 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_created.
unit-cell-a-28: 20:32:02 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_created.
unit-cell-a-30: 20:32:02 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_created.
unit-cell-a-29: 20:32:02 DEBUG unit.cell-a/29.juju-log Emitting Juju event leader_settings_changed.
unit-cell-a-28: 20:32:02 DEBUG unit.cell-a/28.juju-log Emitting Juju event leader_settings_changed.
unit-cell-a-30: 20:32:03 DEBUG unit.cell-a/30.juju-log Emitting Juju event leader_settings_changed.
unit-cell-a-29: 20:32:03 DEBUG unit.cell-a/29.juju-log Emitting Juju event config_changed.
unit-cell-a-28: 20:32:03 DEBUG unit.cell-a/28.juju-log Emitting Juju event config_changed.
unit-cell-a-30: 20:32:03 DEBUG unit.cell-a/30.juju-log Emitting Juju event config_changed.
unit-cell-a-29: 20:32:06 DEBUG unit.cell-a/29.juju-log Emitting Juju event start.
unit-cell-a-28: 20:32:06 DEBUG unit.cell-a/28.juju-log Emitting Juju event start.
unit-cell-a-29: 20:32:07 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-28: 20:32:07 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-30: 20:32:07 DEBUG unit.cell-a/30.juju-log Emitting Juju event start.
unit-cell-a-29: 20:32:07 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-28: 20:32:07 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-30: 20:32:07 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-30: 20:32:07 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-29: 20:32:08 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-28: 20:32:08 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-29: 20:32:08 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-28: 20:32:08 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-30: 20:32:08 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_joined.
unit-cell-a-30: 20:32:09 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-29: 20:32:09 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-28: 20:32:09 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-29: 20:32:09 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-28: 20:32:10 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-30: 20:32:10 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-29: 20:32:10 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-30: 20:32:10 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_changed.
unit-cell-a-28: 20:32:13 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_departed.
unit-cell-a-30: 20:32:13 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_departed.
unit-cell-a-29: 20:32:13 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_departed.
unit-cell-a-30: 20:32:14 DEBUG unit.cell-a/30.juju-log cluster:44: Emitting Juju event cluster_relation_departed.
unit-cell-a-28: 20:32:14 DEBUG unit.cell-a/28.juju-log cluster:44: Emitting Juju event cluster_relation_departed.
unit-cell-a-29: 20:32:14 DEBUG unit.cell-a/29.juju-log cluster:44: Emitting Juju event cluster_relation_departed.
unit-cell-a-30: 20:32:32 DEBUG unit.cell-a/30.juju-log Emitting Juju event leader_elected.

So in the above, we see the peer relation named "cluster" was departed for some reason and it is now showing as "joining":

$ juju status --relations
Model  Controller  Cloud/Region      Version  SLA          Timestamp
test   overlord    juju-lxd/default  3.1.2    unsupported  20:41:42-04:00

App       Version  Status  Scale  Charm             Channel  Rev  Exposed  Message
cell-a             active      3  lxd-cloud-cell              34  no       
region-a           active      3  lxd-cloud-region            12  no

Unit         Workload  Agent  Machine  Public address  Ports  Message
cell-a/28    active    idle   64       10.184.166.81          
cell-a/29    active    idle   65       10.184.166.26          
cell-a/30*   active    idle   66       10.184.166.192         
region-a/6*  active    idle   61       10.184.166.100         
region-a/7   active    idle   62       10.184.166.54          
region-a/8   active    idle   63       10.184.166.254

Machine  State    Address         Inst id         Base          AZ  Message
61       started  10.184.166.100  juju-abb86b-61  ubuntu@22.04      Running
62       started  10.184.166.54   juju-abb86b-62  ubuntu@22.04      Running
63       started  10.184.166.254  juju-abb86b-63  ubuntu@22.04      Running
64       started  10.184.166.81   juju-abb86b-64  ubuntu@22.04      Running
65       started  10.184.166.26   juju-abb86b-65  ubuntu@22.04      Running
66       started  10.184.166.192  juju-abb86b-66  ubuntu@22.04      Running

Relation provider  Requirer          Interface       Type  Message
cell-a:cluster     cell-a:cluster    cell-cluster    peer  joining  
region-a:cluster   region-a:cluster  region-cluster  peer

I'm attaching new logs, charm and snap just in case.

FYI, this is the 3rd times this happens to me after probably a couple hundred deployments (the rev # cannot be trusted as I went through multiple models/contollers). So not a frequent thing but one that really wreak things :/

Revision history for this message

Simon Déziel (sdeziel) wrote on 2023-04-28:

#6

take2-juju-debug.log.gz Edit (187.1 KiB, application/octet-stream)

Revision history for this message

Simon Déziel (sdeziel) wrote on 2023-04-28:

#7

take2-juju-controller.log.gz Edit (54.2 KiB, application/octet-stream)

Revision history for this message

Simon Déziel (sdeziel) wrote on 2023-04-28:

#8

lxd-cloud-cell_ubuntu-22.04-amd64.charm Edit (5.8 MiB, application/octet-stream)

Revision history for this message

Simon Déziel (sdeziel) wrote on 2023-04-28:

#9

lxd-cloud-cell_0+git.6c40e0c_amd64.snap Edit (32.0 MiB, application/octet-stream)

Canonical Juju

<name>_relation_departed is sometimes called randomly

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches