rarp packet will be dropped in flows cause vm connectivity broken after live-migration

Bug #1831404 reported by Yang Li
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
neutron
In Progress
Medium
Yang Li

Bug Description

When we live-migration a VM, at the moment of VM start in destination nodes, the VM will send 5 rarp packets(broadcast) to trigger mac learn for switch, then other same network VMs can send packets to this VM directly, but actually these rarp packets will be dropped in flows of br-int, they won't trigger mac learn for switch, then the connectivity of other VMs to this VM will be broken, they need a broadcast to search the new position of this VM, it will waste some few seconds.

Without live-migration, you can use nping command in VM to simulate rarp packets sent from VM:
nping --arp --arp-type rarp --arp-target-mac <vm-mac> --ether-type rarp <vm-dhcp-host-ip>
For example:
nping --arp --arp-type rarp --arp-target-mac fa:16:3e:02:49:d3 --ether-type rarp 192.168.111.2

And you can use tcpdump to capture these packets, and also you can see these packets are dropped in (table 71 , priority=10, in_port=16), 16 is the tap device port in br-int

# ovs-ofctl show br-int | grep 02:49:d3
 16(tap80be4c89-6d): addr:fe:16:3e:02:49:d3
#
# ovs-ofctl dump-flows br-int | grep table=71 | grep in_port=16 | grep priority=10
 cookie=0xbb76a2777919bc71, duration=1037.306s, table=71, n_packets=13, n_bytes=594, idle_age=12, priority=10,ct_state=-trk,reg5=0x10,in_port=16 actions=drop
#
# tcpdump 'rarp or arp' -i tap80be4c89-6d -nne
tcpdump: WARNING: tap80be4c89-6d: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap80be4c89-6d, link-type EN10MB (Ethernet), capture size 65535 bytes
10:11:11.895958 fa:16:3e:02:49:d3 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 42: Reverse Request who-is fa:16:3e:02:49:d3 tell fa:16:3e:02:49:d3, length 28
10:11:12.896280 fa:16:3e:02:49:d3 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 42: Reverse Request who-is fa:16:3e:02:49:d3 tell fa:16:3e:02:49:d3, length 28
10:11:13.897538 fa:16:3e:02:49:d3 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 42: Reverse Request who-is fa:16:3e:02:49:d3 tell fa:16:3e:02:49:d3, length 28
10:11:14.898815 fa:16:3e:02:49:d3 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 42: Reverse Request who-is fa:16:3e:02:49:d3 tell fa:16:3e:02:49:d3, length 28
10:11:15.900093 fa:16:3e:02:49:d3 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 42: Reverse Request who-is fa:16:3e:02:49:d3 tell fa:16:3e:02:49:d3, length 28

^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel
# ovs-ofctl dump-flows br-int | grep table=71 | grep in_port=16 | grep priority=10
 cookie=0xbb76a2777919bc71, duration=1063.414s, table=71, n_packets=18, n_bytes=804, idle_age=8, priority=10,ct_state=-trk,reg5=0x10,in_port=16 actions=drop

You can see 5 rarp packets are captured by tcpdump, and in the drop flow, the count of n_packets was increased from 13 to 18, this means these 5 rarp packets are dropped by this flow.

Changed in neutron:
assignee: nobody → Yang Li (yang-li)
status: New → In Progress
Yang Li (yang-li)
description: updated
Changed in neutron:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-lib (master)

Reviewed: https://review.opendev.org/661938
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=7d11cd6211a047d73c2a4795e8daa1b6a4851fa6
Submitter: Zuul
Branch: master

commit 7d11cd6211a047d73c2a4795e8daa1b6a4851fa6
Author: Yang Li <email address hidden>
Date: Wed May 29 17:21:53 2019 +0800

    Add rarp protocol for neutron constants

    We should support rarp protocol(0x8035) for live-migration
    A release note is also included.

    Change-Id: I52c115acdb1a8c464f5fc6273f6d8c4d0c6526e9
    Partial-Bug: #1831404

Revision history for this message
norman shen (jshen28) wrote :

Hi, I am curious about the reason why vm sending rarp packets after live-migration? what os is affected? as well as, is there a way to reproduce this issue stably? Any reference related to this bug will be appreciated. Thank you.

Revision history for this message
Yang Li (yang-li) wrote :

It was designed by qemu, when the vm is live-migrate completed, the vm will send 5 rarp packets:
  88 static int announce_self_create(uint8_t *buf,
  89 uint8_t *mac_addr)
  90 {
  91 /* Ethernet header. */
  92 memset(buf, 0xff, 6); /* destination MAC addr */
  93 memcpy(buf + 6, mac_addr, 6); /* source MAC addr */
  94 *(uint16_t *)(buf + 12) = htons(ETH_P_RARP); /* ethertype */
  95
  96 /* RARP header. */
  97 *(uint16_t *)(buf + 14) = htons(ARP_HTYPE_ETH); /* hardware addr space */
  98 *(uint16_t *)(buf + 16) = htons(ARP_PTYPE_IP); /* protocol addr space */
  99 *(buf + 18) = 6; /* hardware addr length (ethernet) */
 100 *(buf + 19) = 4; /* protocol addr length (IPv4) */
 101 *(uint16_t *)(buf + 20) = htons(ARP_OP_REQUEST_REV); /* opcode */
 102 memcpy(buf + 22, mac_addr, 6); /* source hw addr */
 103 memset(buf + 28, 0x00, 4); /* source protocol addr */
 104 memcpy(buf + 32, mac_addr, 6); /* target hw addr */
 105 memset(buf + 38, 0x00, 4); /* target protocol addr */
 106
 107 /* Padding to get up to 60 bytes (ethernet min packet size, minus FCS). */
 108 memset(buf + 42, 0x00, 18);
 109
 110 return 60; /* len (FCS will be added by hardware) */
 111 }

Use some big flavor with high memory, it can be reproduced easily.

Revision history for this message
Yang Li (yang-li) wrote :

This patch showed why the vm send rarp packets after migration.

https://lists.gnu.org/archive/html/qemu-devel/2009-10/msg01457.html

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.