Update permanent ARP entries for allowed_address_pair IPs in DVR Routers

I'm not a big fan of a process running that is snooping on traffic, it's most likely going to cause a performance issue.

Can doing this like the keepalived_state_change code work? It uses "ip monitor" to watch for events and triggers action, and could be modified to look for "neigh" events.

SWe are talking about this piece of code: https://github.com/openstack/neutron/blob/master/neutron/agent/linux/ip_monitor.py#L59

Hi Brian, yes I can take a look at the keepalived_state_change code and see how it works and how it can be used in our case.
But not sure if it can be as such used in our case, since the keepalived in our case is running in side the VM and not in the Namespace.

I don't think that we can use the IP-Monitor for our purpose.
We should definitely come up with a GARP-sniff tool similar to IP-Monitor and then use if for our purpose.
If we think that the performance will be an issue. We should probably come up with a dedicated node, that is doing the sniff and reporting it back to Neutron Server.
That way neutron server can then do an rpc update to all agents to add a permanent entry.

Is there any other way to collect the same information that would be collected by garp-sniffing?
What info needs to be propagated?

Also what about putting a flow rule into the openflow tables that route certain packets to all other compute nodes?

For example, a rule could be added to forward GARP packets to all other "members" of dvr group?

And then you can hang a smarter process off each router namespace and process them however is required. Or make it part of the dvr code.

Also on the performance aspect of sniffing garp packets, the reason a system would suffer performance is if it is processing ALL packets, what if we can filter to only garp packets *before* the monitoring tool gets them.
Its basically the same thing as what a host must do. All hosts have to listen to garp packets, so I am not sure you would get any more traffic than what you would get already.
its just instead of dropping the packets on the floor, you pull them into user space for processing.
Perhaps the term "sniffing" is what we need to avoid.
Again we could put a flow rule that pulls specifically garp packets from an ip address/mac combo and give those packets to a process in user space.

One other simple solution would be to forward a packet ( GARP ) for the MAC's that are configured for Allowed_address_pair to the Ryu controller. The controller can process the packet under the 'packet_in_handler' and then try to create an ARP response entry in the ARP Responder for the MAC and IP.

Before writing the entry probably we should check for the current flows configured in the ARP Responder table and if there is a duplicate entry for the IP/MAC combination, then delete it and rewrite the flows for the GARP packet.

Today we don't have a packet_in_handler for the Ryu controller app.
Similar to this.
@handler.set_ev_cls(ofp_event.EventOFPPacketIn, handler.MAIN_DISPATCHER)
def packet_in_handler(self, ev):
    pkt = packet.Packet(array.array('B', ev.msg.data))
    for p in pkt:
        print p.protocol_name, p

My knowledg is pretty limited in L2 openflow, so if there are any L2 Openflow experts could comment on this, if this would work or not.

Swami I am not sure this last proposal is better in terms of performance than sniffing garp. I don't have all the details and I am new to this problem but have you ever tried instead of setting the NUD to reachable to change the timeout so that arp entries become stale pretty quickly and the GARP can update them? A combination between frequent GARP and entries that get stale quickly might fix this.

Rossella thanks for your feedback. The issue is we are seeing the garp updates the arp entry.But when there is ping to that IP, the router tries to re-arp to confirm the IP and that is were it fails.

Not sure if we can do something similar to ARP Responder to add a dynamic ARP Responder rule to send an ARP reply for the GARP'd MAC.

Adding a rule similar to ARP Responder may not be possible. So the best bet is to forward the GARP packets to an output port (tap port created for this purpose). Then a separate process can listen on the tap port and then parse the packet and provide info to the Network node ( neutron-server) to update the ARP entry on all nodes.

Please provide me your thoughts on this.

If we feel that this process running in the compute node will reduce the performance, then what we should do is probably have a new agent type running this process and then communicating to the Network node. That way we don't need to run this process on all compute nodes.

Related fix proposed to branch: master
Review: https://review.openstack.org/601336

Adding a rule dynamically to the ARP responder is only possible if we can forward the GARP packet to the openflow controller where the openflow controller can process the IN_PACKETS and then make a decision on creating a flow in the ARP responder table.
Is this possible in neutron openflow native drivers today. Any l2 openflow experts can comment on it.
This would be the simplest approach otherwise, we need to forward it to the user space where we process the packet and then send it to Neutron server to take necessary action.

After talking with Miguel Ajo and Daniel Alvarez, plan is to intecept GARP packets and forward to local controller for processing
Swami had a WIP patch that inserted an OF rule to intercept, would need update to send to controller
https://review.openstack.org/#/c/601336/
For ryu look here:
 https://ryu.readthedocs.io/en/latest/ofproto_v1_0_ref.html#packet-in-message
https://ryu.readthedocs.io/en/latest/writing_ryu_app.html?highlight=packet%20in
ovn-controller handles incoming packets (packet-in) as controller here:
https://github.com/openvswitch/ovs/blob/769e6223daf3d6e51963dc3ee938a01fdc71a0d0/ovn/controller/pinctrl.c#L1221
https://github.com/openvswitch/ovs/blob/769e6223daf3d6e51963dc3ee938a01fdc71a0d0/ovn/controller/pinctrl.c#L1121
ODL is tracking this on https://jira.opendaylight.org/browse/NETVIRT-1402
Basically flows will be programmed dynamically when a GARP is recognized by the controller.

Related fix proposed to branch: master
Review: https://review.openstack.org/616272

Related fix proposed to branch: master
Review: https://review.openstack.org/628027

Related fix proposed to branch: master
Review: https://review.openstack.org/651905

Change abandoned by Swaminathan Vasudevan (<email address hidden>) on branch: master
Review: https://review.openstack.org/628027
Reason: Abandoning this patch since I have a better one to address this issue.
https://review.openstack.org/#/c/651905/

Also related to this bug: https://review.opendev.org/#/c/616272/ and https://review.opendev.org/#/c/601336/

Reviewed: https://review.opendev.org/651905
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=52b537ca22b2d7d81a84b2f75de577d8dffee94c
Submitter: Zuul
Branch: master

commit 52b537ca22b2d7d81a84b2f75de577d8dffee94c
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Apr 11 11:12:24 2019 -0700

    DVR: Modify DVR flows to allow ARP requests to hit ARP Responder table

    DVR does the ARP table update through the control plane, and does not
    allow any ARP requests to get out of the node.

    In order to address the allowed address pair VRRP IP issue with DVR,
    we need to add an ARP entry into the ARP Responder table for the
    allowed address pair IP ( which is taken care by the patch in [1])

    This patch adds a rule in the br-int to redirect the packet
    destinated to the router to the actual router-port and also moves
    the arp filtering rule to the tunnel or the physical port based on the
    configuration.

    By adding the above rule it allows the ARP requests to reach the
    ARP Responder table and filters the ARP requests before it reaches
    the physical network or the tunnel.

    [1] https://review.opendev.org/#/c/601336/
    Related-Bug: #1774459

    Change-Id: I3905ea56ca0ff35bdd96c818719e6d63a3eb5a72

Related fix proposed to branch: master
Review: https://review.opendev.org/669938

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/685779

Reviewed: https://review.opendev.org/685779
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=09c4e0e970b02ed992aab762505d17f1decca551
Submitter: Zuul
Branch: stable/stein

commit 09c4e0e970b02ed992aab762505d17f1decca551
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Apr 11 11:12:24 2019 -0700

    DVR: Modify DVR flows to allow ARP requests to hit ARP Responder table

    DVR does the ARP table update through the control plane, and does not
    allow any ARP requests to get out of the node.

    In order to address the allowed address pair VRRP IP issue with DVR,
    we need to add an ARP entry into the ARP Responder table for the
    allowed address pair IP ( which is taken care by the patch in [1])

    This patch adds a rule in the br-int to redirect the packet
    destinated to the router to the actual router-port and also moves
    the arp filtering rule to the tunnel or the physical port based on the
    configuration.

    By adding the above rule it allows the ARP requests to reach the
    ARP Responder table and filters the ARP requests before it reaches
    the physical network or the tunnel.

    [1] https://review.opendev.org/#/c/601336/
    Related-Bug: #1774459

    Change-Id: I3905ea56ca0ff35bdd96c818719e6d63a3eb5a72
    (cherry picked from commit 52b537ca22b2d7d81a84b2f75de577d8dffee94c)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/616272
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Reviewed: https://review.opendev.org/669938
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ea85e39660abaa224c159c42214fdb7042302bea
Submitter: Zuul
Branch: master

commit ea85e39660abaa224c159c42214fdb7042302bea
Author: Brian Haley <email address hidden>
Date: Tue Jul 9 15:50:35 2019 -0400

    Force arp_responder to True when DVR and tunneling enabled

    After [1] and [2], the ARP responder needs to be enabled
    if DVR and tunneling are enabled or ARP will not work. If
    it is False we will log a message and force it to True.

    [1] https://review.opendev.org/#/c/651905/
    [2] https://review.opendev.org/#/c/653883/

    Change-Id: I934062c970effe5194056b0786f84f3246850701
    Related-bug: #1774459

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/700805

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/700988

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/705200

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/705201

Reviewed: https://review.opendev.org/700988
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=dc8c1deeee7ad46958a771567516e91ca52cd156
Submitter: Zuul
Branch: stable/train

commit dc8c1deeee7ad46958a771567516e91ca52cd156
Author: Brian Haley <email address hidden>
Date: Tue Jul 9 15:50:35 2019 -0400

    Force arp_responder to True when DVR and tunneling enabled

    After [1] and [2], the ARP responder needs to be enabled
    if DVR and tunneling are enabled or ARP will not work. If
    it is False we will log a message and force it to True.

    [1] https://review.opendev.org/#/c/651905/
    [2] https://review.opendev.org/#/c/653883/

    Change-Id: I934062c970effe5194056b0786f84f3246850701
    Related-bug: #1774459
    (cherry picked from commit ea85e39660abaa224c159c42214fdb7042302bea)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/705200

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/705201

Change abandoned by Brian Haley (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/700805
Reason: Abandoning based on my last comment, since this would require additional code that isn't present in this branch.

stay tuned

Is someone still working on this bug? Looks to me like it doesn't has any progress since February.

Hi Jorg, Yes, I recently revived patch https://review.opendev.org/#/c/601336/ and I'm working on it currently.

A related patch from Liu:
https://review.opendev.org/c/openstack/neutron/+/768052

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/601336
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Hi Slawek Kaplonski, is someone still working on this bug? Thanks.

We are having trouble to use Octavia LB with HA because of this situation. Octavia keepalived VIP arp record is not updated with gratuitous arp. Details asked to Octavia team but they have nothing to do about this. Octavia Issue: https://storyboard.openstack.org/#!/story/2009765

This is also affecting us with Octavia LB on Xena. Is anyone actively working on this?

We are also having problems with ARP when using octavia LB and DVR routers in Yoga.

We are also running into this issue in both Queens and Victoria (both DVR on OVS). We have been able to work around the GARP issue using the neutron API during a failover, but it would save us a lot of api cycles if the garp was able to trigger updates across the cluster

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/601336
Reason: Please, feel free to restore the patch and rebase the patch.