CMR with Openstack Floating IP Addresses reporting wrong ingress addresses
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Etcd Charm |
Triaged
|
Medium
|
Unassigned | ||
Bug Description
Kubernetes 1.17.4
Juju 2.7.5
Openstack Bionic-Train
MAAS 2.6.2
Testing CMR with a kubernetes control plane (etcd, easyrsa, kubernetes-master, kubeapi-
Kubernetes workers in both Openstack and Bare-metal (provided by MAAS).
Seperate controllers (1 in openstack, 1 in MAAS)
in each model, there are independent containerd and flannel services.
Bare-metal model only has k8s-worker, containerd and flannel.
Using Floating IP in openstack, and the bare-metal env and the openstack FIP are the same subnet/VLAN. There is verified connectivity between the two envs.
When oferring/relating etcd from control in openstack to flannel on bare-metal, flannel gets stuck at waiting on etcd.
Looking at flannel logs we see:
---
2020-04-07 14:20:05 INFO juju-log etcd:3: Invoking reactive handler: reactive/
2020-04-07 14:20:05 INFO juju-log etcd:3: Invoking reactive handler: reactive/
2020-04-07 14:20:23 DEBUG etcd-relation-
2020-04-07 14:20:24 INFO juju-log etcd:3: Unexpected error configuring network. Assuming etcd not ready. Will retry in 20s
---
Performing network-get db against the etcd service we see:
---
$ juju run --unit etcd/0 -- "network-get db"
bind-addresses:
- macaddress: fa:16:3e:0a:40:ee
interfacename: ens2
addresses:
- hostname: ""
address: 192.168.58.145
cidr: 192.168.58.0/24
- hostname: ""
address: 192.168.58.145
cidr: 192.168.58.0/24
- macaddress: 96:d7:c7:ce:62:49
interfacename: fan-252
addresses:
- hostname: ""
address: 252.145.0.1
cidr: 252.0.0.0/8
egress-subnets:
- 192.168.58.145/32
ingress-addresses:
- 192.168.58.145
- 192.168.58.145
- 252.145.0.1
---
The 192.168.58.0/24 addresses are from the internal (vxlan) network of openstack. Etcd should be aware of the floating IP addresses and advertise them. Obviously flannel will never get to that private network.
This could be the offender here:
### juju status from control plane in openstack
$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
default openstack-regionone openstack/RegionOne 2.7.5 unsupported 12:16:15-04:00
App Version Status Scale Charm Store Rev OS Notes
containerd active 3 containerd jujucharms 61 ubuntu
easyrsa 3.0.1 active 1 easyrsa jujucharms 296 ubuntu
etcd 3.3.15 active 3 etcd jujucharms 496 ubuntu
flannel 0.11.0 active 3 flannel jujucharms 468 ubuntu
kubeapi-
kubernetes-master 1.17.4 active 1 kubernetes-master jujucharms 808 ubuntu
kubernetes-
openstack-
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 172.16.7.179 Certificate Authority connected.
etcd/0* active idle 1 172.16.7.180 2379/tcp Healthy with 3 known peers
etcd/1 active idle 2 172.16.7.185 2379/tcp Healthy with 3 known peers
etcd/2 active idle 3 172.16.7.178 2379/tcp Healthy with 3 known peers
kubeapi-
kubernetes-
containerd/1 active idle 172.16.7.186 Container runtime available
flannel/1 active idle 172.16.7.186 Flannel subnet 10.1.48.1/24
kubernetes-
containerd/2 active idle 172.16.7.182 Container runtime available
flannel/2 active idle 172.16.7.182 Flannel subnet 10.1.79.1/24
kubernetes-
containerd/0* active idle 172.16.7.188 Container runtime available
flannel/0* active idle 172.16.7.188 Flannel subnet 10.1.7.1/24
openstack-
Machine State DNS Inst id Series AZ Message
0 started 172.16.7.179 273386c0-
1 started 172.16.7.180 8f1448dd-
2 started 172.16.7.185 90b57213-
3 started 172.16.7.178 a637577c-
4 started 172.16.7.181 f903f10b-
5 started 172.16.7.186 fb740de7-
6 started 172.16.7.182 fd51f143-
7 started 172.16.7.188 9ec4b552-
8 started 172.16.7.183 c255083b-
Offer Application Charm Rev Connected Endpoint Interface Role
easyrsa easyrsa easyrsa 296 1/1 client tls-certificates provider
etcd etcd etcd 496 1/1 db etcd provider
kubeapi-
kubernetes-master kubernetes-master kubernetes-master 808 1/1 kube-control kube-control provider
kubernetes-
Relation provider Requirer Interface Type Message
easyrsa:client etcd:certificates tls-certificates regular
easyrsa:client kubeapi-
easyrsa:client kubernetes-
easyrsa:client kubernetes-
etcd:cluster etcd:cluster etcd peer
etcd:db flannel:etcd etcd regular
etcd:db kubernetes-
kubeapi-
kubeapi-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
openstack-
openstack-
#### juju offers from control plane
$ juju offers
Offer User Relation id Status Endpoint Interface Role Ingress subnets
easyrsa admin 21 joined client tls-certificates provider 172.16.7.0/24
etcd admin 20 joined db etcd provider 172.16.7.0/24
kubeapi-
kubernetes-master admin 22 joined kube-control kube-control provider 172.16.7.0/24
kubernetes-
#### juju status from bare-metal model
$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
k8s-worker jhillman-maas jhillman-maas 2.7.5 unsupported 12:17:27-04:00
SAAS Status Store URL
easyrsa active openstack-regionone admin/default.
etcd active openstack-regionone admin/default.etcd
kubeapi-
kubernetes-master active openstack-regionone admin/default.
kubernetes-
App Version Status Scale Charm Store Rev OS Notes
containerd active 1 containerd jujucharms 61 ubuntu
flannel 0.11.0 maintenance 1 flannel jujucharms 468 ubuntu
kubernetes-
Unit Workload Agent Machine Public address Ports Message
kubernetes-
containerd/0* active idle 10.0.22.5 Container runtime available
flannel/0* maintenance idle 10.0.22.5 Negotiating flannel network subnet.
Machine State DNS Inst id Series AZ Message
0 started 10.0.22.5 agrippa bionic default Deployed
Relation provider Requirer Interface Type Message
easyrsa:client kubernetes-
etcd:db flannel:etcd etcd regular
kubeapi-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
| Changed in charm-etcd: | |
| status: | New → Confirmed |
| George Kraft (cynerva) wrote | #1 |
| Changed in charm-etcd: | |
| importance: | Undecided → High |
| status: | Confirmed → Triaged |
| Changed in charm-etcd: | |
| importance: | High → Medium |
| Changed in charm-etcd: | |
| milestone: | none → 1.25 |
| Changed in charm-etcd: | |
| milestone: | 1.25 → 1.26 |
| Changed in charm-etcd: | |
| milestone: | 1.26 → 1.26+ck1 |
| Changed in charm-etcd: | |
| milestone: | 1.26+ck1 → 1.26+ck2 |
| Changed in charm-etcd: | |
| milestone: | 1.26+ck2 → 1.27 |
| Kevin W Monroe (kwmonroe) wrote | #2 |
| Changed in charm-etcd: | |
| milestone: | 1.27 → 1.28 |
| Changed in charm-etcd: | |
| milestone: | 1.28 → 1.28+ck1 |
| Changed in charm-etcd: | |
| milestone: | 1.28+ck1 → 1.29 |
Confirmed. The etcd charm, when calling network-get for the db relation, does not pass in relation IDs. This means it does not fully support cross-model relations.
This won't be easy to fix. The etcd charm will need to call network-get with relation IDs for the db relation. The charm might observe different ingress addresses for each relation, so it will need to be able to send a different address to each relation. The etcd interface doesn't easily support this - it's still using the old RelationBase with scope=GLOBAL, meaning that the etcd units need to communicate with eachother via the cluster relation and agree on a single connection string before sending that off to db relations.
Most likely, fixing this will involve first updating interface-etcd to use the Endpoint class instead of RelationBase.