Canonical Juju

migrated sidecar unit agents are still talking to the old controller

Bug #2008756 reported by Yang Kelvin Liu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Critical
Unassigned
Canonical Juju 3.1.2

Bug Description

juju migrate k2:t1 k1

juju models
Controller: k1

Model Cloud/Region Type Status Units Access Last connection
controller microk8s/localhost kubernetes available 1 admin just now
t1 microk8s/localhost kubernetes available 2 admin 7 minutes ago

juju status
Model Controller Cloud/Region Version SLA Timestamp
t1 k1 microk8s/localhost 3.1.1 unsupported 17:07:23+11:00

App Version Status Scale Charm Channel Rev Address Exposed Message
prometheus-k8s 2.33.5 error 0/2 prometheus-k8s latest/stable 103 10.152.183.65 no StatefulSet.apps "prometheus-k8s" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'rep...

Unit Workload Agent Address Ports Message
prometheus-k8s/0 unknown lost 10.1.97.223 agent lost, see 'juju show-status-log prometheus-k8s/0'
prometheus-k8s/1 unknown lost 10.1.97.223 agent lost, see 'juju show-status-log prometheus-k8s/1'

2023-02-28T06:08:21.317Z [container-agent] 2023-02-28 06:08:21 INFO juju.api apiclient.go:702 connection established to "wss://controller-service.controller-k2.svc.cluster.local:17070/model/83cdee97-d48d-4bf9-8e6f-e6b9f162ad06/api"
2023-02-28T06:08:21.318Z [container-agent] 2023-02-28 06:08:21 ERROR juju.worker.dependency engine.go:695 "api-caller" manifold worker returned unexpected error: [83cdee] "unit-prometheus-k8s-0" cannot open api: unknown model: "83cdee97-d48d-4bf9-8e6f-e6b9f162ad06" (model not found)

Tags: k8s
Revision history for this message
Yang Kelvin Liu (kelvin.liu) wrote :

https://pastebin.ubuntu.com/p/JTRvBGxc94/

Changed in juju:
importance: Undecided → Critical
milestone: none → 3.1.1
Yang Kelvin Liu (kelvin.liu)
Changed in juju:
status: New → Triaged
Yang Kelvin Liu (kelvin.liu)
Changed in juju:
status: Triaged → In Progress
Yang Kelvin Liu (kelvin.liu)
tags: added: k8s
Revision history for this message
Yang Kelvin Liu (kelvin.liu) wrote (last edit ):

This wrong address issue has gone on 3.1 now.
The caas provisioner worker running on the target controller updates the new controller address to the <app-name>-application-config secret and also updates the controller.juju.is/id annotation in the statefulset which causes the sidecar pod gets restarted to read the new addr from ENV. So we are ok now,
The reason why had this issue before was because this bug https://bugs.launchpad.net/juju/+bug/2008744 causes the provisioner can never update the statefulset successfully to trigger sidecar pods to restart and also it can't update the new addr into the secret(They are in the same Ensure() call).

So I cannot reproduce this bug after we fixed https://bugs.launchpad.net/juju/+bug/2008744.
I would say it's been fixed by https://github.com/juju/juju/pull/15264

Changed in juju:
status: In Progress → Fix Committed
Harry Pidcock (hpidcock)
Changed in juju:
milestone: 3.1.1 → 3.1.2
Canonical Juju QA Bot (juju-qa-bot)
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.