| 2020-09-24 16:08:38 |
Nobuto Murata |
bug |
|
|
added bug |
| 2020-09-24 16:10:57 |
Nobuto Murata |
bug |
|
|
added subscriber Canonical Field High |
| 2020-09-24 20:11:23 |
John A Meinel |
summary |
network-get returns /32 address and units cannot talk to each other |
network-get starts returning flannel address instead of host nic address |
|
| 2020-09-24 20:11:51 |
John A Meinel |
juju: importance |
Undecided |
High |
|
| 2020-09-24 20:11:51 |
John A Meinel |
juju: status |
New |
Incomplete |
|
| 2020-09-24 23:35:45 |
Nobuto Murata |
juju: status |
Incomplete |
New |
|
| 2020-09-28 01:21:08 |
Nobuto Murata |
attachment added |
|
reproducer.yaml https://bugs.launchpad.net/juju/+bug/1897115/+attachment/5414828/+files/reproducer.yaml |
|
| 2020-09-28 01:26:41 |
Nobuto Murata |
description |
juju version: 2.8.3-bionic-amd64
provider: vSphere
Charmed Kubernetes deployment doesn't settle and is not usable with "waiting: Waiting to retry addon deployment" in kubernetes-master. The root cause is Juju believes that /32 addresses from an vxlan interface of flannel in kubernetes-master units are the ones other services should use instead of the "main" /26 addresses the units have. By the nature of /32 in IPv4, nobody can reach to it.
kubernetes-master has the "main" and flannel/32 address:
$ juju run --unit kubernetes-master/leader -- ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
ens192 UP 133.XX.XX.XX/26 fe80::XXXX:XXXX:XXXX:XXXX/64
flannel.1 UNKNOWN 10.1.75.0/32 fe80::YYYY:YYYY:YYYY:YYYY/64
network-get returns 10.1.75.0/32 which is not reachable from other units:
$ juju run --unit kubernetes-master/leader -- network-get kube-api-endpoint --ingress-address
10.1.75.0
Other services such as kubeapi-loadbalancer writes the following config, then Nginx cannot reach to the backend servers.
$ cat /etc/nginx/sites-enabled/apilb
upstream target_service {
server 10.1.75.0:6443;
server 10.1.8.0:6443;
}
And there is no way to specify which subnet to be used explicitly because network spaces are not supported in vSphere provider as far as I'm concerned.
$ juju spaces
cannot list spaces: spaces not supported (not supported)
ERROR cannot list spaces: spaces not supported (not supported)
Here is the full output of network-get, and it doesn't contain the expected /26 address at all.
$ juju run --unit kubernetes-master/leader -- network-get kube-api-endpoint
bind-addresses:
- macaddress: fa:73:41:53:bd:2c
interfacename: flannel.1
addresses:
- hostname: ""
address: 10.1.75.0
cidr: 10.1.75.0/32
egress-subnets:
- 10.1.75.0/32
ingress-addresses:
- 10.1.75.0
FWIW, "primary-network" is already set in model-config. |
juju version: 2.8.3-bionic-amd64
provider: vSphere
Charmed Kubernetes deployment doesn't settle and is not usable with "waiting: Waiting to retry addon deployment" in kubernetes-master. The root cause is Juju believes that /32 addresses from an vxlan interface of flannel in kubernetes-master units are the ones other services should use instead of the "main" /26 addresses the units have. By the nature of /32 in IPv4, nobody can reach to it.
How to reproduce:
1. Bootstrap with local/LXD profier
2. Define a separate bridge as follows:
$ lxc network create lxdbr-rfc6598 \
ipv4.address=100.64.0.1/26 \
ipv4.dhcp.ranges=100.64.0.11-100.64.0.62 \
ipv4.nat=true \
ipv6.address=none \
3. Define a profile with it:
$ lxc profile create juju-rfc6598
$ lxc profile edit juju-rfc6598 <<EOF
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr-rfc6598
type: nic
EOF
4. Create a model matching with the name of the profile:
$ juju add-model rfc6598
5. Make sure the logging level is DEBUG or more, and deploy:
$ juju model-config logging-config
<root>=DEBUG
$ juju deploy ./reproducer.yaml
reproducer.yaml:
https://bugs.launchpad.net/juju/+bug/1897115/+attachment/5414828/+files/reproducer.yaml
kubernetes-master has the "main" and flannel/32 address:
$ juju run --unit kubernetes-master/leader -- ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
ens192 UP 133.XX.XX.XX/26 fe80::XXXX:XXXX:XXXX:XXXX/64
flannel.1 UNKNOWN 10.1.75.0/32 fe80::YYYY:YYYY:YYYY:YYYY/64
network-get returns 10.1.75.0/32 which is not reachable from other units:
$ juju run --unit kubernetes-master/leader -- network-get kube-api-endpoint --ingress-address
10.1.75.0
Other services such as kubeapi-loadbalancer writes the following config, then Nginx cannot reach to the backend servers.
$ cat /etc/nginx/sites-enabled/apilb
upstream target_service {
server 10.1.75.0:6443;
server 10.1.8.0:6443;
}
And there is no way to specify which subnet to be used explicitly because network spaces are not supported in vSphere provider as far as I'm concerned.
$ juju spaces
cannot list spaces: spaces not supported (not supported)
ERROR cannot list spaces: spaces not supported (not supported)
Here is the full output of network-get, and it doesn't contain the expected /26 address at all.
$ juju run --unit kubernetes-master/leader -- network-get kube-api-endpoint
bind-addresses:
- macaddress: fa:73:41:53:bd:2c
interfacename: flannel.1
addresses:
- hostname: ""
address: 10.1.75.0
cidr: 10.1.75.0/32
egress-subnets:
- 10.1.75.0/32
ingress-addresses:
- 10.1.75.0
FWIW, "primary-network" is already set in model-config. |
|
| 2020-09-28 01:31:50 |
Nobuto Murata |
description |
juju version: 2.8.3-bionic-amd64
provider: vSphere
Charmed Kubernetes deployment doesn't settle and is not usable with "waiting: Waiting to retry addon deployment" in kubernetes-master. The root cause is Juju believes that /32 addresses from an vxlan interface of flannel in kubernetes-master units are the ones other services should use instead of the "main" /26 addresses the units have. By the nature of /32 in IPv4, nobody can reach to it.
How to reproduce:
1. Bootstrap with local/LXD profier
2. Define a separate bridge as follows:
$ lxc network create lxdbr-rfc6598 \
ipv4.address=100.64.0.1/26 \
ipv4.dhcp.ranges=100.64.0.11-100.64.0.62 \
ipv4.nat=true \
ipv6.address=none \
3. Define a profile with it:
$ lxc profile create juju-rfc6598
$ lxc profile edit juju-rfc6598 <<EOF
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr-rfc6598
type: nic
EOF
4. Create a model matching with the name of the profile:
$ juju add-model rfc6598
5. Make sure the logging level is DEBUG or more, and deploy:
$ juju model-config logging-config
<root>=DEBUG
$ juju deploy ./reproducer.yaml
reproducer.yaml:
https://bugs.launchpad.net/juju/+bug/1897115/+attachment/5414828/+files/reproducer.yaml
kubernetes-master has the "main" and flannel/32 address:
$ juju run --unit kubernetes-master/leader -- ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
ens192 UP 133.XX.XX.XX/26 fe80::XXXX:XXXX:XXXX:XXXX/64
flannel.1 UNKNOWN 10.1.75.0/32 fe80::YYYY:YYYY:YYYY:YYYY/64
network-get returns 10.1.75.0/32 which is not reachable from other units:
$ juju run --unit kubernetes-master/leader -- network-get kube-api-endpoint --ingress-address
10.1.75.0
Other services such as kubeapi-loadbalancer writes the following config, then Nginx cannot reach to the backend servers.
$ cat /etc/nginx/sites-enabled/apilb
upstream target_service {
server 10.1.75.0:6443;
server 10.1.8.0:6443;
}
And there is no way to specify which subnet to be used explicitly because network spaces are not supported in vSphere provider as far as I'm concerned.
$ juju spaces
cannot list spaces: spaces not supported (not supported)
ERROR cannot list spaces: spaces not supported (not supported)
Here is the full output of network-get, and it doesn't contain the expected /26 address at all.
$ juju run --unit kubernetes-master/leader -- network-get kube-api-endpoint
bind-addresses:
- macaddress: fa:73:41:53:bd:2c
interfacename: flannel.1
addresses:
- hostname: ""
address: 10.1.75.0
cidr: 10.1.75.0/32
egress-subnets:
- 10.1.75.0/32
ingress-addresses:
- 10.1.75.0
FWIW, "primary-network" is already set in model-config. |
juju version: 2.8.3-bionic-amd64
provider: vSphere
Charmed Kubernetes deployment doesn't settle and is not usable with "waiting: Waiting to retry addon deployment" in kubernetes-master. The root cause is Juju believes that /32 addresses from an vxlan interface of flannel in kubernetes-master units are the ones other services should use instead of the "main" /26 addresses the units have. By the nature of /32 in IPv4, nobody can reach to it.
How to reproduce:
1. Bootstrap with local/LXD profier
2. Define a separate bridge as follows:
$ lxc network create lxdbr-rfc6598 \
ipv4.address=100.64.0.1/26 \
ipv4.dhcp.ranges=100.64.0.11-100.64.0.62 \
ipv4.nat=true \
ipv6.address=none \
3. Define a profile with it:
$ lxc profile create juju-rfc6598
$ lxc profile edit juju-rfc6598 <<EOF
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr-rfc6598
type: nic
EOF
4. Create a model matching with the name of the profile:
$ juju add-model rfc6598
5. Make sure the logging level is DEBUG or more, and deploy:
$ juju model-config logging-config
<root>=DEBUG
$ juju deploy ./reproducer.yaml
reproducer.yaml:
https://bugs.launchpad.net/juju/+bug/1897115/+attachment/5414828/+files/reproducer.yaml
6. Once the model settles, wait for 10 or 20 minutes until "observed network config updated" event is triggered.
> DEBUG juju.worker.machiner machiner.go:181 observed network config updated
[Actual]
The unit will have two IP addresses, one is /26 and the other is /32 on flannel.1. Juju's network-get will select /32 as the ingress-address.
$ juju show-machine 0
...
network-interfaces:
eth0:
ip-addresses:
- 100.64.0.48
mac-address: 00:16:3e:51:24:68
gateway: 100.64.0.1
is-up: true
flannel.1:
ip-addresses:
- 10.1.27.0
mac-address: 7e:f7:e2:9a:11:b6
is-up: true
$ juju run --unit kubernetes-master/0 -- ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
flannel.1 UNKNOWN 10.1.27.0/32 fe80::7cf7:e2ff:fe9a:11b6/64
eth0@if31 UP 100.64.0.48/26 fe80::216:3eff:fe51:2468/64
$ juju run --unit kubernetes-master/0 -- network-get kube-api-endpoint --ingress-address
10.1.27.0
$ juju run --unit kubernetes-master/0 -- network-get kube-api-endpoint
bind-addresses:
- macaddress: 7e:f7:e2:9a:11:b6
interfacename: flannel.1
addresses:
- hostname: ""
address: 10.1.27.0
cidr: 10.1.27.0/32 <<<<<<<<<<
egress-subnets:
- 10.1.27.0/32
ingress-addresses:
- 10.1.27.0
[Expected]
Juju will pick up /26 one instead of /32 which is not reachable from other units at all. |
|
| 2020-09-28 01:43:07 |
Nobuto Murata |
bug |
|
|
added subscriber Canonical Field Critical |
| 2020-09-28 01:43:12 |
Nobuto Murata |
removed subscriber Canonical Field High |
|
|
|
| 2020-09-28 14:47:40 |
Nobuto Murata |
bug |
|
|
added subscriber Canonical Field High |
| 2020-09-28 14:55:08 |
Nobuto Murata |
removed subscriber Canonical Field Critical |
|
|
|
| 2020-09-29 14:46:13 |
Joseph Phillips |
juju: assignee |
|
Joseph Phillips (manadart) |
|
| 2020-09-29 14:46:17 |
Joseph Phillips |
juju: status |
New |
In Progress |
|
| 2020-10-22 13:49:17 |
Pen Gale |
juju: milestone |
|
3.0.0 |
|
| 2020-10-22 13:49:24 |
Pen Gale |
juju: status |
In Progress |
Triaged |
|
| 2022-10-22 02:20:16 |
Canonical Juju QA Bot |
juju: milestone |
3.0.0 |
3.0.1 |
|
| 2022-11-15 09:48:31 |
Canonical Juju QA Bot |
juju: milestone |
3.0.1 |
3.0.2 |
|
| 2022-11-16 09:16:14 |
Canonical Juju QA Bot |
juju: milestone |
3.0.2 |
3.0.3 |
|
| 2023-02-09 03:43:22 |
Canonical Juju QA Bot |
juju: milestone |
3.0.3 |
3.0.4 |
|
