Ubuntu Cloud Archive

  1. Series wallaby
  2. Bug #1940043
  3. Comment #11

Comment 11 for bug 1940043

Revision history for this message
Frode Nordahl (fnordahl) wrote :

Package versions before we start:
$ juju run --application ovn-central 'dpkg -l |grep ovn'
- Stdout: |
    ii ovn-central 20.03.2-0ubuntu0.20.04.4 amd64 OVN central components
    ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components
  UnitId: ovn-central/0
- Stdout: |
    ii ovn-central 20.03.2-0ubuntu0.20.04.4 amd64 OVN central components
    ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components
  UnitId: ovn-central/1
- Stdout: |
    ii ovn-central 20.03.2-0ubuntu0.20.04.4 amd64 OVN central components
    ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components
  UnitId: ovn-central/2

$ juju run --application ovn-chassis 'dpkg -l |grep ovn'
- Stdout: |
    ii neutron-ovn-metadata-agent 2:16.4.2-0ubuntu4 all Neutron is a virtual network service for Openstack - OVN metadata agent
    ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components
    ii ovn-host 20.03.2-0ubuntu0.20.04.4 amd64 OVN host components
  UnitId: ovn-chassis/0
- Stdout: |
    ii neutron-ovn-metadata-agent 2:16.4.2-0ubuntu4 all Neutron is a virtual network service for Openstack - OVN metadata agent
    ii ovn-common 20.03.2-0ubuntu0.20.04.4 amd64 OVN common components
    ii ovn-host 20.03.2-0ubuntu0.20.04.4 amd64 OVN host components
  UnitId: ovn-chassis/1

Ping running instances:
$ ping 10.78.95.55
PING 10.78.95.55 (10.78.95.55) 56(84) bytes of data.
64 bytes from 10.78.95.55: icmp_seq=1 ttl=63 time=1.80 ms
64 bytes from 10.78.95.55: icmp_seq=2 ttl=63 time=1.22 ms
64 bytes from 10.78.95.55: icmp_seq=3 ttl=63 time=1.06 ms
...

$ ping 10.78.95.162
PING 10.78.95.162 (10.78.95.162) 56(84) bytes of data.
64 bytes from 10.78.95.162: icmp_seq=1 ttl=63 time=1.08 ms
64 bytes from 10.78.95.162: icmp_seq=2 ttl=63 time=0.545 ms
64 bytes from 10.78.95.162: icmp_seq=3 ttl=63 time=0.516 ms
...

Ensure OVN DNS interception/resolution is enabled and working:
ubuntu@zaza-neutrontests-ins-1:~$ dig zaza-neutrontests-ins-2 @10.78.95.1
...
;; ADDITIONAL SECTION:
zaza-neutrontests-ins-2. 3600 IN A 192.168.0.180

Ensure ovn-controllers picks up version mismatch prior to upgrade:

Note that the backported version mismatch handling code does not have the
additional version mismatch check in the incremental processing engine that
later versions have, this means that we need to ensure the main loop version
mismatch check has run prior to allowing northd to fill database tables after
an upgrade. We will just have to deal with this in the charms and/or as part
of upgrade documentation.

Force the version mismatch to happen before we actually perform the upgrade to
ensure the ovn-controller does not make any mistakes.

$ juju run --application ovn-central 'systemctl stop ovn-northd; systemctl mask ovn-northd'
$ juju run --unit ovn-central/0 'ovn-sbctl set sb-global . options:northd_internal_version="20.03.2-2.7.0-42.1"'
$ juju run --application ovn-chassis 'tail -1 /var/log/ovn/ovn-controller.log'
- Stdout: |
    2022-11-06T11:04:26.305Z|00017|main|WARN|controller version - 20.03.2-2.7.0-42.0 mismatch with northd version - 20.03.2-2.7.0-42.1
  UnitId: ovn-chassis/0
- Stdout: |
    2022-11-06T11:04:26.272Z|00021|main|WARN|controller version - 20.03.2-2.7.0-42.0 mismatch with northd version - 20.03.2-2.7.0-42.1
  UnitId: ovn-chassis/1

Confirm OVN DNS interception/resolution is still working:
ubuntu@zaza-neutrontests-ins-1:~$ dig zaza-neutrontests-ins-2 @10.78.95.1
...
;; ADDITIONAL SECTION:
zaza-neutrontests-ins-2. 3600 IN A 192.168.0.180

Upgrade packages on central units:
$ juju config ovn-central ovn-source=cloud:focal-ovn-22.03

$ juju run --application ovn-central 'systemctl unmask ovn-northd; systemctl restart ovn-northd'

Confirm instances are still responding:
64 bytes from 10.78.95.55: icmp_seq=1167 ttl=63 time=1.31 ms
64 bytes from 10.78.95.55: icmp_seq=1168 ttl=63 time=1.06 ms
64 bytes from 10.78.95.55: icmp_seq=1169 ttl=63 time=1.07 ms
...

64 bytes from 10.78.95.162: icmp_seq=1145 ttl=63 time=0.569 ms
64 bytes from 10.78.95.162: icmp_seq=1146 ttl=63 time=0.564 ms
64 bytes from 10.78.95.162: icmp_seq=1147 ttl=63 time=0.937 ms
...

Confirm OVN DNS interception/resolution is still working:
ubuntu@zaza-neutrontests-ins-1:~$ dig zaza-neutrontests-ins-2 @10.78.95.1
...
;; ADDITIONAL SECTION:
zaza-neutrontests-ins-2. 3600 IN A 192.168.0.180

Upgrade packages on chassis units:
$ juju config ovn-chassis ovn-source=cloud:focal-ovn-22.03

Collect instance ping statistics after completing the upgrade:
64 bytes from 10.78.95.55: icmp_seq=1299 ttl=63 time=1.02 ms
64 bytes from 10.78.95.55: icmp_seq=1300 ttl=63 time=1.04 ms
64 bytes from 10.78.95.55: icmp_seq=1301 ttl=63 time=1.01 ms
^C
--- 10.78.95.55 ping statistics ---
1301 packets transmitted, 1293 received, 0.614912% packet loss, time 1301821ms
rtt min/avg/max/mdev = 0.871/1.296/28.365/1.310 ms

64 bytes from 10.78.95.162: icmp_seq=1264 ttl=63 time=0.642 ms
64 bytes from 10.78.95.162: icmp_seq=1265 ttl=63 time=0.711 ms
64 bytes from 10.78.95.162: icmp_seq=1266 ttl=63 time=0.564 ms
^C
--- 10.78.95.162 ping statistics ---
1266 packets transmitted, 1261 received, 0.394945% packet loss, time 1292577ms
rtt min/avg/max/mdev = 0.434/0.677/28.644/0.921 ms