Unable to contact to IPv6 instance using ml2 ovs with ovs 2.16
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Ubuntu Cloud Archive |
New
|
Undecided
|
Unassigned | ||
| Xena |
Fix Released
|
Undecided
|
Unassigned | ||
| Yoga |
Fix Released
|
Undecided
|
Unassigned | ||
| neutron |
Invalid
|
Undecided
|
Unassigned | ||
| openvswitch (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
| Impish |
Won't Fix
|
Undecided
|
Unassigned | ||
| Jammy |
Fix Released
|
Undecided
|
Unassigned | ||
| Kinetic |
Fix Released
|
High
|
Unassigned | ||
Bug Description
Connectivity is fine with OVS 2.15 but after upgrading ovs, connectivity is lost to remote units over ipv6. The traffic appears to be lost while being processed by the openflow firewall associated with br-int.
The description below uses connectivity between Octavia units and amphora to illustrate the issue but I don't think this issue is related to Octavia.
OS: Ubuntu Focal
OVS: 2.16.0-
Kernel: 5.4.0-100-generic
With a fresh install of xena or after an upgrade of OVS from 2.15 (wallaby) to 2.16 (xena) connectivity from the octavia units to the amphora is broken.
* Wallaby works as expected
* Disabling port security on the octavia units octavia-
* The flows on br-int and br-tun are the same after the upgrade from 2.15 to 2.16
* Manually inserting permissive flows into the br-int flow table also restores connectivity.
* Testing environment is Openstack on top of Openstack.
Text below is reproduced here https:/
Below is reproduction of the issue first deploying wallaby to validate connectivity before upgrading openvswitch.
Amphora:
$ openstack loadbalancer amphora list
+------
| id | loadbalancer_id | status | role | lb_network_ip | ha_ip |
+------
| 30afe97a-
| 61e66eff-
+------
$ openstack router show lb-mgmt -c name -c interfaces_info
+------
| Field | Value |
+------
| interfaces_info | [{"port_id": "191a2d27-
| name | lb-mgmt |
+------
Looking at ports on that subnet there is a port for each of the octavia units (named octavia-
each of the amphora listed above and a port for the lb-mgmt router.
$ openstack port list | grep 8b4307a7-
| 0943521f-
| 160b8854-
| 191a2d27-
| 2428b1d4-
| 2ea37e19-
| 76742ab6-
| ffb3d106-
The ports attached to the octavia units have port security enabled:
$ openstack port show octavia-
+------
| Field | Value |
+------
| device_owner | neutron:
| id | 76742ab6-
| name | octavia-
| port_security_
| security_group_ids | 04582e3a-
+------
$ openstack security group rule list 04582e3a-
+------
| ID | IP Protocol | Ethertype | IP Range | Port Range | Direction | Remote Security Group | Remote Address Group |
+------
| 3bde542e-
| 89c4f2ed-
| a26b3608-
| bfc54f3f-
+------
Connectivity between the octavia units and the amphora is working:
$ ping -c1 fc00:92e3:
PING fc00:92e3:
64 bytes from fc00:92e3:
--- fc00:92e3:
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.820/3.
$ ping -c1 fc00:92e3:
PING fc00:92e3:
64 bytes from fc00:92e3:
--- fc00:92e3:
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.123/4.
$ nc -zvw2 fc00:92e3:
Connection to fc00:92e3:
$ nc -zvw2 fc00:92e3:
Connection to fc00:92e3:
Take a dump of the flows before upgrade:
sudo ovs-ofctl dump-flows br-int --no-stats > br-int-
sudo ovs-ofctl dump-flows br-tun --no-stats > br-tun-
Switch apt sources to xena:
$ sudo sed -i -e 's/wallaby/xena/' /etc/apt/
$ sudo apt update
$ apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.15.0-
Candidate: 2.16.0-
Version table:
2.
500 http://
*** 2.15.0-
100 /var/lib/
2.
500 http://
2.
500 http://
2.
500 http://
Upgrade openvswitch-switch and restart services:
$ sudo apt install openvswitch-switch
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
openvswitch-
Suggested packages:
openvswitch-doc
The following packages will be upgraded:
openvswitch-
3 upgraded, 0 newly installed, 0 to remove and 79 not upgraded.
Need to get 2930 kB of archives.
After this operation, 285 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://
Get:2 http://
Get:3 http://
Fetched 2930 kB in 1s (3485 kB/s)
(Reading database ... 92182 files and directories currently installed.)
Preparing to unpack .../python3-
Unpacking python3-openvswitch (2.16.0-
Preparing to unpack .../openvswitch
Unpacking openvswitch-common (2.16.0-
Preparing to unpack .../openvswitch
Unpacking openvswitch-switch (2.16.0-
Setting up python3-openvswitch (2.16.0-
Setting up openvswitch-common (2.16.0-
Setting up openvswitch-switch (2.16.0-
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for systemd (245.4-4ubuntu3.15) ...
$ sudo systemctl restart ovs-vswitchd.
$ sudo systemctl restart neutron-
$ sudo systemctl restart neutron-
Retest connectivity:
$ ping -c1 fc00:92e3:
PING fc00:92e3:
--- fc00:92e3:
1 packets transmitted, 0 received, 100% packet loss, time 0ms
$ ping -c1 fc00:92e3:
PING fc00:92e3:
--- fc00:92e3:
1 packets transmitted, 0 received, 100% packet loss, time 0ms
$ nc -zvw2 fc00:92e3:
nc: connect to fc00:92e3:
$ nc -zvw2 fc00:92e3:
nc: connect to fc00:92e3:
Check for changes in flows:
sudo ovs-ofctl dump-flows br-int --no-stats > br-int-
sudo ovs-ofctl dump-flows br-tun --no-stats > br-tun-
$ diff <(sed -e 's!cookie=
23,25d22
< cookie=COOKIE, table=71, priority=
< cookie=COOKIE, table=71, priority=
< cookie=COOKIE, table=71, priority=
29c26,28
< cookie=COOKIE, table=71, priority=
---
> cookie=COOKIE, table=71, priority=
> cookie=COOKIE, table=71, priority=
> cookie=COOKIE, table=71, priority=
30a30
> cookie=COOKIE, table=71, priority=
32d31
< cookie=COOKIE, table=71, priority=
33a33
> cookie=COOKIE, table=71, priority=
37d36
< cookie=COOKIE, table=71, priority=
38a38
> cookie=COOKIE, table=71, priority=
$ diff <(sed -e 's!cookie=
2d1
< cookie=COOKIE, priority=
3a3
> cookie=COOKIE, priority=
20,21d19
< cookie=COOKIE, table=20, priority=
< cookie=COOKIE, table=20, priority=
23c21,22
< cookie=COOKIE, table=20, priority=
---
> cookie=COOKIE, table=20, priority=
> cookie=COOKIE, table=20, priority=
25a25
> cookie=COOKIE, table=20, priority=
27,28d26
< cookie=COOKIE, table=20, hard_timeout=300, priority=
< cookie=COOKIE, table=20, hard_timeout=300, priority=
30a29,30
> cookie=COOKIE, table=20, hard_timeout=300, priority=
> cookie=COOKIE, table=20, hard_timeout=300, priority=
The only changes are the cookie values and the order that dumpflow has written them out, the flows are actually unchanged
$ diff <(sed -e 's!cookie=
$
$ diff <(sed -e 's!cookie=
$
Connectivity can be restored by disabling port security on the ocatvia ports:
$ openstack port set --no-security-group --disable-
$ ping -c1 fc00:92e3:
PING fc00:92e3:
64 bytes from fc00:92e3:
--- fc00:92e3:
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.955/2.
$ ping -c1 fc00:92e3:
PING fc00:92e3:
64 bytes from fc00:92e3:
--- fc00:92e3:
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.113/2.
$ nc -zvw2 fc00:92e3:
Connection to fc00:92e3:
$ nc -zvw2 fc00:92e3:
Connection to fc00:92e3:
Re-enable port security
$ openstack port set --security-group 04582e3a-
Connectivity can be also be restored by manually installing permissive flows into the flows associated with br-int:
$ sudo ovs-ofctl add-flow br-int table=0,
$ sudo ovs-ofctl add-flow br-int table=0,
$ ping -c1 fc00:92e3:
PING fc00:92e3:
64 bytes from fc00:92e3:
--- fc00:92e3:
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.230/4.
$ sudo ovs-ofctl add-flow br-int table=0,
$ sudo ovs-ofctl add-flow br-int table=0,
$ nc -zvw2 fc00:92e3:
Connection to fc00:92e3:
| description: | updated |
| Liam Young (gnuoy) wrote | #1 |
| Liam Young (gnuoy) wrote | #2 |
| Changed in neutron: | |
| status: | New → Invalid |
| Liam Young (gnuoy) wrote | #3 |
| Pedro Victor Lourenço Fragola (pedrovlf) wrote | #4 |
| tags: | added: sts |
| Frode Nordahl (fnordahl) wrote | #5 |
| affects: | openvswitch → cloud-archive |
| Changed in openvswitch (Ubuntu Kinetic): | |
| status: | New → Triaged |
| importance: | Undecided → High |
| status: | Triaged → In Progress |
| status: | In Progress → Fix Committed |
| status: | Fix Committed → In Progress |
| Changed in openvswitch (Ubuntu Kinetic): | |
| status: | In Progress → Fix Released |
| Changed in openvswitch (Ubuntu Jammy): | |
| status: | New → Fix Released |
| Changed in openvswitch (Ubuntu Impish): | |
| status: | New → Won't Fix |
| Felipe Reyes (freyes) wrote | #6 |
Capturing a ping packet from the src port o-hm0 and tracing it through the flows shows that it should work (below replicated here https:/
(While running ping fc00:92e3:
$ sudo tcpdump -i o-hm0 dst fc00:92e3:
$ ovs-pcap icmp.pcap > icmp.hex
$ PACKET=$(head -n1 icmp.hex)
$ sudo ovs-appctl ofproto/trace br-int in_port="o-hm0" $PACKET
Flow: icmp6,in_
bridge("br-int")
----------------
0. priority 0, cookie 0x9f446960f3350392
goto_table:60
60. in_port=3, priority 100, cookie 0x9f446960f3350392
set_
set_
resubmit(,71)
71. ipv6,reg5=
ct(
drop
-> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 72.
-> Sets the packet to an untracked state, and clears all the conntrack fields.
Final flow: icmp6,reg5=
ode=0
Megaflow: recirc_
Datapath actions: ct(zone=
=======
recirc(0x3f) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
=======
Flow: recirc_
,nw_ecn=
bridge("br-int")
----------------
thaw
Resuming from table 72
72. ct_state=
resubmit(,73)
73. ct_state=
ct(
drop
-> Sets the packet to an untracked state, and clears all the conntrack fields.
resubmit(,91)
91. priority 1, cookie 0x9f446960f3350392
resubmit(,94)
94. priority 1, cookie ...