Inability to add cross model relation

This bug occurred on the same juju controller as bug https://bugs.launchpad.net/juju/+bug/1952796

Relevant artifacts for this bug...

Juju controller SOS reports:
https://private-fileshare.canonical.com/~sbparke/sosreport-juju-1-Juju-CMR-Controller-0-2021-11-30-npxesvi.tar.xz
https://private-fileshare.canonical.com/~sbparke/sosreport-juju-3-Juju-CMR-Controller-1-2021-11-30-olgynuc.tar.xz
https://private-fileshare.canonical.com/~sbparke/sosreport-juju-2-Juju-CMR-Controller-2-2021-11-30-lahomhg.tar.xz

Crash Dumps for snap-store and LMA models:
https://private-fileshare.canonical.com/~sbparke/juju-crashdump-59a7cfdd-f754-4641-aebd-26c072ca29eb-lmaModel.tar.xz
https://private-fileshare.canonical.com/~sbparke/juju-crashdump-29162c9a-5244-45b6-ab1e-07927bd614b8-snap-storeModel.tar.xz

Controller database:
https://private-fileshare.canonical.com/~sbparke/sanitizedMongoDB.tgz

Quick comment on the relation count

Offer Application Charm Rev Connected Endpoint Interface Role
grafana grafana grafana 51 3/3 dashboards grafana-dashboard requirer
graylog-beats graylog graylog 50 1/1 beats elastic-beats provider <<<<--- 1/1 relations ?

When you relate a consuming application to the offer, there is only the one relation (aka connection) displayed; this is independent of how many units may be involved. It's no different to in-model relations in "juju status --relations" - only one item is shown per relation even if many units are involved.

Depending on what the exact issue is, it could be related to exactly what relation data is passed between the offering and consuming charm. We've seen in the past that charms can select an inappropriate address for example. We'd need to know exactly what's misconfigured, how it's misconfigured, where the config comes from, what's missing or incorrect etc etc to narrow down is it a juju issue, is it a charm issue etc etc.

Ok that's good to know.

We are rebuilding this deployment but I will try and replicate in the lab and provide the requested data.

Ok I have been able to replicate this in the lab with nagios and nrpe cross model relations.

The proper hook sequence seems to be:
  relation-created
  relation-joined
  relation-changed

It seems that relation-joined is not being fired for cross model relations when deleting and adding relations after deployment. It does seem to fire on initial deployment of the charms.

Below you can see the deployment, followed by the remove-relation and add-relation actions.

Here is an nrpe cross model relation remove/add action:

fgrep relation unit-nrpe-0.log.back.CrossModel
2021-12-05 04:55:14 INFO juju.worker.uniter.operation runhook.go:155 skipped "general-info-relation-created" hook (missing)
2021-12-05 04:55:14 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-created" hook (missing)
2021-12-05 04:55:31 INFO juju.worker.uniter.operation runhook.go:152 ran "monitors-relation-joined" hook (via explicit, bespoke hook script)
2021-12-05 04:55:36 INFO juju.worker.uniter.operation runhook.go:152 ran "monitors-relation-changed" hook (via explicit, bespoke hook script)
2021-12-05 04:55:42 INFO juju.worker.uniter.operation runhook.go:152 ran "general-info-relation-joined" hook (via explicit, bespoke hook script)
2021-12-05 04:55:47 INFO juju.worker.uniter.operation runhook.go:152 ran "general-info-relation-changed" hook (via explicit, bespoke hook script)
2021-12-05 04:56:57 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-departed" hook (missing)
2021-12-05 04:56:58 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-broken" hook (missing)
2021-12-05 04:57:45 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-created" hook (missing)

--- Missing relation-joined hook ---

Here is the same action within the same model:

fgrep relation unit-nrpe-0.log.back.SameModel | fgrep -v WARNING
2021-12-05 06:34:39 INFO juju.worker.uniter.operation runhook.go:155 skipped "nrpe-external-master-relation-created" hook (missing)
2021-12-05 06:35:04 INFO juju.worker.uniter.operation runhook.go:152 ran "nrpe-external-master-relation-joined" hook (via explicit, bespoke hook script)
2021-12-05 06:35:14 INFO juju.worker.uniter.operation runhook.go:152 ran "nrpe-external-master-relation-changed" hook (via explicit, bespoke hook script)
2021-12-05 06:35:58 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-created" hook (missing)
2021-12-05 06:36:05 INFO juju.worker.uniter.operation runhook.go:152 ran "monitors-relation-joined" hook (via explicit, bespoke hook script)
2021-12-05 06:36:13 INFO juju.worker.uniter.operation runhook.go:152 ran "monitors-relation-changed" hook (via explicit, bespoke hook script)
2021-12-05 06:38:01 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-departed" hook (missing)
2021-12-05 06:38:04 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-broken" hook (missing)
2021-12-05 06:39:14 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-created" hook (missing)
2021-12-05 06:39:21 INFO juju.worker.uniter.operation runhook.go:152 ran "monitors-relation-joi...

I have replicated this bug with nrpe/nagios and graylog/filebeats. I do not think this is an issue with charm code.
It seems that juju is not triggering the relation-joined hook for cross model relations.

Please find more files for reference under private-fileshare.canonical.com:~sbparke/Bug-1952796-dump.tgz

tar tf Bug-1952796-dump.tgz
dump/
dump/nrpe-test/
dump/nrpe-test/unit-nrpe-0.log.back.SameModel <--- nrpe logs from remove/add relation in same model
dump/nrpe-test/client.yaml <--- client bundle
dump/nrpe-test/unit-nrpe-0.log.back.CrossModel <--- remove/add relation in cross model
dump/nrpe-test/lma.yaml <-- lma bundle

dump/filebeat-test/
dump/filebeat-test/client.yaml <<-- client bundle
dump/filebeat-test/unit-graylog-0.log.backup <--- graylog unit logs
dump/filebeat-test/controller-logsink.log.back <-- juju controller
dump/filebeat-test/unit-filebeat-0.log.backup <--- filebeat unit
dump/filebeat-test/ControllerDBDump.yaml <<-- juju controller DB dump
dump/filebeat-test/controller-machine-0.log.back
dump/filebeat-test/graylog.yaml <<--- lma bundle
dump/filebeat-test/unit-graylog-1.log.back
dump/filebeat-test/graylog-machine-1.log.back
dump/filebeat-test/unit-graylog-2.log.back

What I'm seeing is the lack of the relation-joined hooks being fired when adding and removing cross model relations. The hook does seem to fire within the same model.

Hopefully you can deploy the client and lma bundles and then remove and add cross model relations to see the issue in your lab.

Please replicate with the following commands:

From the files under dump/filebeat-test provided above you will find the bundles used below

Steps:

# deploy graylog
juju deploy ./graylog.yaml

# wait for model to settle

# deploy the client bundle
juju deploy ./client.yaml

# wait for model to settle
# filebeat should be in ready state

#remove
juju remove-relation filebeat graylog

# wait for model to settle
# filebeat will go into waiting state

# add relation
juju add-relation filebeat graylog

# filebeat never comes out of wait state
# tail logs for filebeat agent

This PR allows the steps above to complete successfully. Tested on AWS.

https://github.com/juju/juju/pull/13559