ICMPv6 Neighbor Advertisement packets from VM's link-local address dropped by OVS

So I wasn't able to reproduce this yet locally, although I've only been able to test on a private network which I assumed would show the same issue. But I figured I'd add a note with at least what I saw.

First, the OVS firewall will add a flow for the link-local address for IPv6 NA traffic, it was added in https://review.opendev.org/c/openstack/neutron/+/783743 which shows it backported to basically all releases.

First I tested with ML2/OVS and the OVS Firewall driver, testing was with devstack on the master branch (Bobcat). After booting a VM I saw the following flows for NA:

$ sudo ovs-appctl dpif/dump-flows br-int | grep 136 | grep fe80
recirc_id(0),in_port(9),ct_state(-trk),eth(src=fa:16:3e:aa:fc:06,dst=fa:16:3e:ae:2e:fe),eth_type(0x86dd),ipv6(src=fe80::f816:3eff:feaa:fc06,proto=58,frag=no),key32(00 00/00 00),icmpv6(type=136), packets:0, bytes:0, used:never, actions:6
recirc_id(0),in_port(9),ct_state(-trk),eth(src=fa:16:3e:aa:fc:06,dst=33:33:00:00:00:01),eth_type(0x86dd),ipv6(src=fe80::f816:3eff:feaa:fc06,proto=58,frag=no),key32(00 00/00 00),icmpv6(type=136), packets:2, bytes:172, used:9.348s, actions:push_vlan(vid=1,pcp=0),1,pop_vlan,4,5,6

So no actions=drop rules. Pings from the qrouter namespace to the LL address worked fine.

Then I tried with ML2/OVN, same setup. After booting a VM I saw the following flows for NA:

$ sudo ovs-ofctl dump-flows br-int | grep 136 | grep 35d9
 cookie=0x2467f8c7, duration=248.112s, table=9, n_packets=9, n_bytes=774, idle_age=136, priority=90,ipv6,reg14=0x4,metadata=0x1,dl_src=fa:16:3e:6a:35:d9,ipv6_src=fe80::f816:3eff:fe6a:35d9 actions=resubmit(,10)
 cookie=0x0, duration=248.110s, table=10, n_packets=0, n_bytes=0, idle_age=248, priority=90,icmp6,reg14=0x4,metadata=0x1,dl_src=fa:16:3e:6a:35:d9,nw_ttl=255,icmp_type=136,icmp_code=0,nd_target=fe80::f816:3eff:fe6a:35d9 actions=conjunction(1093703813,1/2)
 cookie=0x0, duration=248.110s, table=10, n_packets=0, n_bytes=0, idle_age=248, priority=90,icmp6,reg14=0x4,metadata=0x1,dl_src=fa:16:3e:6a:35:d9,nw_ttl=255,icmp_type=136,icmp_code=0,nd_target=fd2c:af9f:6196:0:f816:3eff:fe6a:35d9 actions=conjunction(1093703813,1/2)
 cookie=0x9e65c598, duration=248.113s, table=48, n_packets=2, n_bytes=204, idle_age=136, priority=90,ipv6,reg15=0x4,metadata=0x1,dl_dst=fa:16:3e:6a:35:d9,ipv6_dst=fe80::f816:3eff:fe6a:35d9 actions=resubmit(,49)

Again, no actions=drop rules, and pings worked fine.

Things could be different on a provider network, when I booted a similar VM using a shared network it had other issues, just don't have time at the moment to debug that.

This was all running with OVN 22.03.2 from the Ubuntu cloud archives.

Hi Brian, thanks for your reply.

I've done some more testing and it turns out I can reproduce it using two VMs instances on a common regular tenant network (Geneve encap). They're both in the same "default" security group, so all traffic between them should be allowed. The subnet object/ports does have GUA IPv6 prefix/ports assigned.

The symptoms are the same as for the VM on the VLAN provider network:

On the hypervisor hosting the VM with LLA fe80::f816:3eff:fe9b:a982:
--------------------------------------------------------------------

$ sudo ovs-ofctl dump-flows br-int | egrep 'icmp_type=136.*:(9a5e|a982)'
 cookie=0x8a1ff59b, duration=1423.329s, table=74, n_packets=0, n_bytes=0, idle_age=1423, priority=90,icmp6,reg14=0x9,metadata=0x26,nw_ttl=225,icmp_type=136,icmp_code=0,nd_target=fe80::f816:3eff:fe9b:a982,nd_tll=00:00:00:00:00:00 actions=load:0->NXM_NX_REG10[12]
 cookie=0x8a1ff59b, duration=1423.329s, table=74, n_packets=0, n_bytes=0, idle_age=1423, priority=90,icmp6,reg14=0x9,metadata=0x26,nw_ttl=225,icmp_type=136,icmp_code=0,nd_target=fe80::f816:3eff:fe9b:a982,nd_tll=fa:16:3e:9b:a9:82 actions=load:0->NXM_NX_REG10[12]

$ sudo ovs-appctl dpif/dump-flows br-int | grep 136 | grep fe80
recirc_id(0),in_port(5),eth(src=fa:16:3e:9b:a9:82),eth_type(0x86dd),ipv6(src=fe80::f816:3eff:fe9b:a982,proto=58,hlimit=255,frag=no),icmpv6(type=136,code=0),nd(target=fe80::f816:3eff:fe9b:a982,tll=00:00:00:00:00:00), packets:260, bytes:20280, used:5.170s, actions:drop

On the hypervisor hosting the VM with LLA fe80::f816:3eff:fead:9a5e:
--------------------------------------------------------------------

$ sudo ovs-ofctl dump-flows br-int | egrep 'icmp_type=136.*:(9a5e|a982)'
 cookie=0x89bf2021, duration=1434.895s, table=74, n_packets=0, n_bytes=0, idle_age=1434, priority=90,icmp6,reg14=0x8,metadata=0x26,nw_ttl=225,icmp_type=136,icmp_code=0,nd_target=fe80::f816:3eff:fead:9a5e,nd_tll=00:00:00:00:00:00 actions=load:0->NXM_NX_REG10[12]
 cookie=0x89bf2021, duration=1434.895s, table=74, n_packets=0, n_bytes=0, idle_age=1434, priority=90,icmp6,reg14=0x8,metadata=0x26,nw_ttl=225,icmp_type=136,icmp_code=0,nd_target=fe80::f816:3eff:fead:9a5e,nd_tll=fa:16:3e:ad:9a:5e actions=load:0->NXM_NX_REG10[12]

$ sudo ovs-appctl dpif/dump-flows br-int | grep 136 | grep fe80
recirc_id(0),in_port(5),eth(src=fa:16:3e:ad:9a:5e),eth_type(0x86dd),ipv6(src=fe80::f816:3eff:fead:9a5e,dst=fe80::/ffff::,proto=58,hlimit=255,frag=no),icmpv6(type=136,code=0),nd(target=fe80::f816:3eff:fead:9a5e,tll=00:00:00:00:00:00), packets:260, bytes:20280, used:4.587s, actions:drop

'ndisc6 ${LLA_OF_VM1} eth0' on VM2 and vice versa consistently fails. 'tcpdump -ni eth0 icmp6' on the target VM shows the NS towards the solicited-node multicast address show up and be answered with an NA, but the latter does not show up in a corresponding tcpdump session on the source VM.

I see that I am running a newer OVN version (23.03) than you do (22.03.2) which I suppose might be relevant.

I'll do some more digging and let you know if I find out anything else.

why no one notify my report https://bugs.launchpad.net/neutron/+bug/2031085

I figured out how to do some more tracing/debugging of the vanishing NA packets, and it seems to be a mismatch between what actually happens in OVS (drop) and what OVN thinks should happening (forwarding). I'll walk you through it below.

1) First I capture an ICMPv6 NA packet that is dropped from the :a982 VM:

$ tcpdump -XXvpnei tap8bdc141e-fe -Q in -c 1 icmp6[0] = 136
dropped privs to tcpdump
tcpdump: listening on tap8bdc141e-fe, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:22:08.178527 fa:16:3e:9b:a9:82 > fa:16:3e:ad:9a:5e, ethertype IPv6 (0x86dd), length 78: (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::f816:3eff:fe9b:a982 > fe80::f816:3eff:fead:9a5e: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is fe80::f816:3eff:fe9b:a982, Flags [solicited]
        0x0000: fa16 3ead 9a5e fa16 3e9b a982 86dd 6000 ..>..^..>.....`.
        0x0010: 0000 0018 3aff fe80 0000 0000 0000 f816 ....:...........
        0x0020: 3eff fe9b a982 fe80 0000 0000 0000 f816 >...............
        0x0030: 3eff fead 9a5e 8800 ad9d 4000 0000 fe80 >....^....@.....
        0x0040: 0000 0000 0000 f816 3eff fe9b a982 ........>.....
1 packet captured
1 packet received by filter
0 packets dropped by kernel

2) Then I determine the ofport attribute for the VM-facing tap interface:

$ ovs-vsctl get Interface tap8bdc141e-fe ofport
15

3) Using the packet hex string + the ofport attribute found above, I can now do a OVS trace, which can then be run through OVN detrace to be annotated with additional information:

$ ovs-appctl ofproto/trace br-int in_port=15 fa163ead9a5efa163e9ba98286dd6000000000183afffe80000000000000f8163efffe9ba982fe80000000000000f8163efffead9a5e8800ad9d40000000fe80000000000000f8163efffe9ba982 | ovn-detrace (…)
Flow: icmp6,in_port=15,vlan_tci=0x0000,dl_src=fa:16:3e:9b:a9:82,dl_dst=fa:16:3e:ad:9a:5e,ipv6_src=fe80::f816:3eff:fe9b:a982,ipv6_dst=fe80::f816:3eff:fead:9a5e,ipv6_label=0x00000,nw_tos=0,nw_ecn=0,nw_ttl=255,nw_frag=no,icmp_type=136,icmp_code=0,nd_target=fe80::f816:3eff:fe9b:a982,nd_sll=00:00:00:00:00:00,nd_tll=00:00:00:00:00:00

bridge("br-int")
----------------
0. in_port=15, priority 100, cookie 0x8a1ff59b
set_field:0x46->reg13
set_field:0x45->reg11
set_field:0x44->reg12
set_field:0x26->metadata
set_field:0x9->reg14
resubmit(,8)
  * Logical datapath: "neutron-7f71277c-aae0-4037-8392-2398a8be9929" (9c1b45e9-0135-4301-b7a8-4281e843e306)
  * Port Binding: logical_port "8bdc141e-fed6-4100-bf80-95a36c8a9b97", tunnel_key 9, chassis-name "d48404f8-e1b9-5184-bdda-bd7ce0ca1d58", chassis-str "node08"
8. metadata=0x26, priority 50, cookie 0x59f248ee
set_field:0/0x1000->reg10
resubmit(,73)
  * Logical datapaths:
  * "neutron-b7a0e814-603b-4a7e-9bc2-c5a14752fcbf" (0486544d-1050-408d-936f-aaaa96c71df7) [ingress]
  * "neutron-4f34ef66-41ad-4393-a0e7-c46981d55740" (0ef0124f-4c01-481b-9bd8-e0cbdd60883b) [ingress]
  * "neutron-50ba6904-3111-43a3-8fe4-d2bb6d6d120c" (14741cc3-fe87-4605-b17a-72db8c9f46f9) [ingress]
  * "neutron-75c4c616-b079-42f2-bf62-c4049a16ac31" (155bf97f-55d0-40f3-9246-731983eb9fc5) [ingress]
  * "neutron-54578d02-3847-442c-adc0-1a7ad144ef05" (25e90264-92e7-4941-aafa-96b7f08...

Hi Tore,

Sorry for the slow response just busy with a lot of things. Thanks for reproducing this with a tenant network, make it easier to debug (for me).

I will try and reproduce this with a later OVN version, should be able to trigger a build from source in devstack. But I am still curious why you have a 'drop' rule as I don't see neutron adding one, unless ovn-controller is doing that.

-Brian

Hi Brian, no worries, I'm just glad someone is taking a look at all!

I have some good news: the issue is reproducible using Devstack branch stable/2023.1 on Rocky 9. I have a such a Devstack instance up and running right now that exhibits the issue, where the two VMs cannot resolve each others' LLAs because the NAs are dropped.

When Devstack is running on Ubuntu 22.04, on the other hand, it works fine, both using the stable/2023.1 and master branches.

I have not been able to test Devstack master on Rocky 9 due to bug #2031639.

I would be happy to give you access to all these Devstack instances, so you can see it for yourself. If you would like that, just send me an SSH pubkey and I will add it to the authorized_keys files. (You could also send it to me on IRC - I'm tore@#openstack-neutron.)

Tore

A workaround for bug #2031639 was found, so now I also have a Devstack master testbed on Rocky 9. It also exhibits this bug. So to summarise my testing:

Platform | Devstack version | Neutron version | OVN version | Exhibits bug
Ubuntu-22.04 | stable/2023.1 | 22.0.3.dev43 | 22.03.2-0ubuntu0.22.04.1 | No
Ubuntu-22.04 | master | 23.0.0.0b3.dev261 | 22.03.2-0ubuntu0.22.04.1 | No
Rocky-9 | stable/2023.1 | 22.0.3.dev43 | 23.03.0-69.el9s | Yes
Rocky-9 | master | 23.0.0.0b3.dev263 | 23.03.0-69.el9 | Yes

The offer to give you access to all of the above testbeds stands, I'll leave them running for now.

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/892012

So I hate to say, but only being broken on Rocky 9 with their OVN packages points at a distro issue.

Is Rocky 9 close to Centos 9? If so we can create a test patch and run the experimental jobs, there is a neutron-ovn-tempest-ovs-master-centos-9-stream job that will build OVS and OVN from source on Centos 9.

https://review.opendev.org/c/openstack/neutron/+/892012

We're seeing it on AlmaLinux-9 as well (our production rig), so it is definitively not a Rocky-9 specific issue. It is possible that it comes from RHEL-9 (both AlmaLinux and Rocky are RHEL clones). Both AlmaLinux and Rocky are related to CentOS-9, as CentOS-9 is upstream for RHEL-9.

The RPM packages for all of the above comes from https://www.rdoproject.org/.

I personally believe it is more likely that it is an upstream OVN issue, and that the reason it does not (yet!) occur on Ubuntu-22.04 is that the Deb packages it uses contain an older OVN version than the RPMs Rocky/AlmaLinux/RHEL/CentOS provided by RDO Project.

Yeah, it might be a but in OVN.

I found the changelog for the code for 23.03 in Centos:

https://cbs.centos.org/koji/buildinfo?buildID=49874

And looking at that, Ihar made some changes in the IPv6 code regarding stateless security groups, I wonder if something got broken? I pinged him on irc to see if he had any thoughts on it, his nick is ihrachys.

The experimental pipeline is currently broken trying to build OVN and OVS from source, so testing manually seems the only way at the moment.

Nailed it:

[rocky@devstack-2 ovn]$ git bisect bad
8cab00bdb581f714dcfdc4fb08affe0319c211a6 is the first bad commit
commit 8cab00bdb581f714dcfdc4fb08affe0319c211a6
Author: Numan Siddique <email address hidden>
Date: Thu May 19 11:17:39 2022 -0400

    ovn-controller: Add OF rules for port security.

    ovn-controller will now generate OF rules for in port security and
    out port security checks in OF tables - 73, 74 and 75. These flows
    will be added if a port binding has port security defined in the
    Port_Binding.Port_Security column which is newly added in this patch.

    The idea of this patch is to program these OF rules directly within
    the ovn-controller instead of ovn-northd generating logical flows.
    This helps in reducing the numnber of logical flows overall in the
    Southbound database.

    Upcoming patches will add the necessary OVN actions which ovn-northd
    can make use of.

    Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=2078927
    Suggested-by: Dumitru Ceara <email address hidden>
    Acked-by: Mark Michelson <email address hidden>
    Signed-off-by: Numan Siddique <email address hidden>
    Signed-off-by: Mark Michelson <email address hidden>

 controller/binding.c | 78 ++++-
 controller/binding.h | 23 +-
 controller/lflow.c | 792 ++++++++++++++++++++++++++++++++++++++++++-
 controller/lflow.h | 4 +
 controller/ovn-controller.c | 21 +-
 include/ovn/actions.h | 4 +
 include/ovn/logical-fields.h | 1 +
 ovn-sb.ovsschema | 7 +-
 ovn-sb.xml | 15 +
 tests/ovn.at | 288 ++++++++++++++++
 10 files changed, 1199 insertions(+), 34 deletions(-)

Reproduced and bisected this using devstack master (21eac99e4e342108d7905f64c3e5474b70c9273f) on Rocky 9, then building and installing OVN and OVS from source into /opt/ovn and overriding the ExecStop/ExecStart settings in ovn-controller/ovn-northd.service to use /opt/ovn/share/ovn/scripts/ovn-ctl instead of the default from the RDO RPM.

@Tore, thanks for bisecting! I was actually looking at the exact same commit just now because it seems like the potential culprit of the regression in 23.03 from just reading the code. So - good coincidence!

Could you please share `dump-flows br-int` for the good and bad trees of OVN? We can then compare the exact tables where port_security is implemented to see where the difference hides.

Thanks in advance.

I will also try to pull Numan here since he's the author of the commit and may have an idea what happens. (I need to check when he's back in office though.)

I've now attached output from dump-flows from the first bad commit (8cab00bdb) and from the last good commit (c0224a14f). While the dumps were performed, the two VMs were constantly trying soliciting each others link-local and ULA addresses.

If you want access to the test VM, just send me an SSH public key.

@Hi Tore,

regarding your bisect, I think the exact commit you identified may expose failing behavior because it lacks the full implementation of the feature, so it doesn't create any flows in table=7[3-5] at all.

Could you please repeat your dump-flows with commits included up to 94974a02bfd5b24b89a6a7471133730fd7fa1356? These will show the final set of flows as created by ovn-controller (and not through Logical_Flows).

Thank you!

Certainly, @Ihar. I've attached updated dumps now. The observed behaviour is the same as with 8cab00bdb.

For what it's worth, the contents of the tables you mentioned appear to be identical too (when stripping out the duration, packet and byte counters:

[tore:/tmp] $ sed -n '/table=7\(3\|4\|5\)/s/, duration=.*, priority=//p' ovs-ofctl_dump-flows_br-int.8cab00bdb.txt | md5sum
0eecf2a4f76f6ee954153f4764964ad6 -
[tore:/tmp] $ sed -n '/table=7\(3\|4\|5\)/s/, duration=.*, priority=//p' ovs-ofctl_dump-flows_br-int.94974a02b.txt | md5sum
0eecf2a4f76f6ee954153f4764964ad6 -

It is only the last good commit that doesn't create any flows in those tables:

[tore:/tmp] $ grep -c table=7 ovs-ofctl_dump-flows_br-int.c0224a14f.txt
0

Change abandoned by "Brian Haley <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/892012
Reason: Tore found the change that broke this in OVN

Could you please attach the relevant Logical_Flow records from NB for the affected port? And also, while at it, please show the output of `openstack port show` and `ovn-nbctl list Logical_Switch_Port` and `ovn-sbctl list Port_Binding` for the relevant ports. Thank you!

I wonder if you have relevant port security Logical_Flows created for the port in SB database...

Nevermind, I think I know what's going on, I've found a bug in the new table=7* flows: the higher priority flows that are supposed to allow NAs for the ipv6 addresses match against nw_ttl=225 instead of nw_ttl=255, so legit NAs are not matching and fall back down to the default drop flow.

I will post the fix upstream this week. I will link it here.

Fix: https://patchwork<email address hidden>/

Tore, if you can validate it helps, I would appreciate it.

Good catch! Matching on TTL=225 is clearly wrong.

However, I tried applying your patch on top of current main branch, but the nw_ttl=225 flows still show up in the output of ovs-ofctl dump-flows br-int anyway, and it still does not work.

Reading the OVN code I do not understand how this could be, unless these nw_ttl=225 flows are persistently cached somewhere and thus survive upgrades and restarts of ovn-controller and ovn-northd. (Which would be a problem for the production rig as well.)

In trying to figure this out and make sure I started over completely from scratch, I've managed to completely hose my devstack...now the OVN services are complaining about «X table in OVN_{North,South}bound database lacks Y column (database needs upgrade?)» and «OVN_{North,South}bound database lacks X table». Tried to recover from this with

sudo /opt/ovn/bin/ovsdb-tool create /var/lib/ovn/ovnnb_db.db /opt/ovn/share/ovn/ovn-nb.ovsschema
sudo /opt/ovn/bin/ovsdb-tool create /var/lib/ovn/ovnsb_db.db /opt/ovn/share/ovn/ovn-sb.ovsschema

…but no cigar. I'll see if I can figure out how to recover, if not I'll try to re-establish a brand new devstack and see if I can make that work.

Okay, figured out what was going wrong. Even though I had installed OVN into /opt/ovn, and started it using "/opt/ovn/share/ovn/scripts/ovn-ctl start_{controller,northd}", the ovn-ctl script went on to starting the OVN binaries installed to /usr/bin instead for some dumb reason. After moving the /usr/bin binaries out of the way, ovn-ctl started the right binaries from /opt/ovn instead.

That made me able to confirm that the patch works as expected. With it, the nw_ttl=225 flows are replaced with nw_ttl=255 ones, and IPv6 neighbour discovery between the VMs works again. Great job!

By the way, I think this patch should be cherry-picked onto the stable branches that contain 8cab00bdb581 ("ovn-controller: Add OF rules for port security.").

Yes, the patch will be cherry-picked into all affected (supported) branches as part of the master merge, by whoever is merging it in master.

And thanks for confirming the patch works. I don't know what the procedure for the LP bug for neutron would be, considering that the bug is in OVN. Do we close it?

Let's keep it open until the patch merges and we can list the OVN versions that have it. It's already assigned to me so can update then.

From Mark

```
Thanks Ales and Ihar,

I applied this change to all branches from main back to 22.09.

```

This will be fixed in next upstream releases.

Closing as it's fixed in OVN tree.