Juniper Openstack

DPDK Compute cannot communicate to the gateway in datapath- cannot ping the gateway

  1. Series trunk
  2. Bug #1751124
Bug #1751124 reported by kratna
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.1
Fix Committed
Critical
Naveen N
Juniper Openstack r4.1.1.0
Trunk
Fix Committed
Critical
Naveen N
Juniper Openstack r5.0.0

Bug Description

During RHOPS10 + Contrail 4.1 deployment ,

The DPDK compute is unable to ping the gateway in datapath.

ip a

<none>
[root@overcloud-contraildpdk-1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 0c:c4:7a:81:a4:a4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.53/24 brd 192.168.30.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fe81:a4a4/64 scope link
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 0c:c4:7a:81:a4:a5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ec4:7aff:fe81:a4a5/64 scope link
       valid_lft forever preferred_lft forever
4: enp2s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 90:e2:ba:da:fa:30 brd ff:ff:ff:ff:ff:ff
5: enp2s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 90:e2:ba:da:fa:31 brd ff:ff:ff:ff:ff:ff
7: ens7f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 0c:c4:7a:b7:2c:fb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ec4:7aff:feb7:2cfb/64 scope link
       valid_lft forever preferred_lft forever
9: vhost0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 0c:c4:7a:b7:2c:fa brd ff:ff:ff:ff:ff:ff
    inet 10.88.0.81/24 brd 10.88.0.255 scope global vhost0
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:feb7:2cfa/64 scope link
       valid_lft forever preferred_lft forever
10: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether c2:34:32:7d:ee:ce brd ff:ff:ff:ff:ff:ff
11: vlan213@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 0c:c4:7a:81:a4:a5 brd ff:ff:ff:ff:ff:ff
    inet 10.88.2.30/24 brd 10.88.2.255 scope global vlan213
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fe81:a4a5/64 scope link
       valid_lft forever preferred_lft forever
12: vlan211@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 0c:c4:7a:81:a4:a5 brd ff:ff:ff:ff:ff:ff
    inet 10.88.1.29/24 brd 10.88.1.255 scope global vlan211
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fe81:a4a5/64 scope link
       valid_lft forever preferred_lft forever
13: vlan215@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 0c:c4:7a:81:a4:a5 brd ff:ff:ff:ff:ff:ff
    inet 10.88.4.30/24 brd 10.88.4.255 scope global vlan215
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fe81:a4a5/64 scope link
       valid_lft forever preferred_lft forever
14: vlan214@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 0c:c4:7a:81:a4:a5 brd ff:ff:ff:ff:ff:ff
    inet 10.88.3.32/24 brd 10.88.3.255 scope global vlan214
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fe81:a4a5/64 scope link
       valid_lft forever preferred_lft forever
[root@overcloud-contraildpdk-1 ~]#

vif --list

[root@overcloud-contraildpdk-1 ~]# vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
       Proxy=MAC Requests Proxied Always, Er=Etree Root, Mn=Mirror without Vlan Tag

vif0/0 PCI: 0000:00:00.0 (Speed 10000, Duplex 1)
            Type:Physical HWaddr:0c:c4:7a:b7:2c:fa IPaddr:0.0.0.0
            Vrf:0 Mcast Vrf:65535 Flags:TcL3L2VpEr QOS:-1 Ref:16
            RX device packets:23236 bytes:3279686 errors:0
            RX port packets:10726 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:10726 bytes:1529400 errors:0
            TX packets:3377 bytes:310706 errors:0
            Drops:3013
            TX port packets:3371 errors:6
            TX device packets:66412 bytes:8025938 errors:0

vif0/1 PMD: vhost0
            Type:Host HWaddr:0c:c4:7a:b7:2c:fa IPaddr:0.0.0.0
            Vrf:0 Mcast Vrf:65535 Flags:L3DEr QOS:-1 Ref:14
            RX device packets:3235 bytes:304742 errors:0
            RX queue packets:3235 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:3235 bytes:304742 errors:0
            TX packets:7713 bytes:1234126 errors:0
            Drops:0
            TX queue packets:7713 errors:0
            TX device packets:7713 bytes:1234126 errors:0

vif0/2 Socket: unix
            Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0.0.0.0
            Vrf:65535 Mcast Vrf:65535 Flags:L3Er QOS:-1 Ref:3
            RX port packets:142 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:142 bytes:12212 errors:0
            TX packets:686 bytes:71344 errors:0
            Drops:0

[root@overcloud-contraildpdk-1 ~]#

[root@overcloud-contraildpdk-1 ~]# /opt/contrail/bin/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=uio_pci_generic unused=ixgbe

Network devices using kernel driver
===================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp2s0f0 drv=ixgbe unused=uio_pci_generic
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp2s0f1 drv=ixgbe unused=uio_pci_generic
0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=ens7f1 drv=ixgbe unused=uio_pci_generic
0000:04:00.0 'I350 Gigabit Network Connection' if=eno1 drv=igb unused=uio_pci_generic *Active*
0000:04:00.1 'I350 Gigabit Network Connection' if=eno2 drv=igb unused=uio_pci_generic

Other network devices
=====================
<none>
[root@overcloud-contraildpdk-1 ~]#

[root@overcloud-contraildpdk-1 ~]# ps -aef | grep dpdk
root 11009 10998 99 01:38 ? 1-06:30:35 /usr/bin/contrail-vrouter-dpdk --no-daemon --vdev eth_bond_bond0,mode=4,xmit_policy=l34,socket_id=0,mac=0c:c4:7a:b7:2c:fa,slave=0000:03:00.0 --socket-mem 1024 1024
root 24342 24266 0 09:07 pts/0 00:00:00 grep --color=auto dpdk
[root@overcloud-contraildpdk-1 ~]#

[root@overcloud-contraildpdk-1 ~]# cat /etc/contrail/contrail-vrouter-agent.conf
[DEFAULT]
platform=dpdk
physical_interface_address=0000:00:00.0
physical_interface_mac=0c:c4:7a:b7:2c:fa
physical_uio_driver=uio_pci_generic
log_file=/var/log/contrail/contrail-vrouter-agent.log
log_level=SYS_NOTICE
log_local=1
[VIRTUAL-HOST-INTERFACE]
physical_interface=bond0
name=vhost0
gateway=10.88.0.254
[SERVICE-INSTANCE]
netns_command=/bin/opencontrail-vrouter-netns
[root@overcloud-contraildpdk-1 ~]#

Jeba Paulaiyan (jebap)
tags: added: dpdk
kratna (kratna)
Changed in juniperopenstack:
assignee: Raja Sivaramakrishnan (raja-u) → kratna (kratna)
assignee: kratna (kratna) → nobody
assignee: nobody → Raja Sivaramakrishnan (raja-u)
summary: - DPDK Compute cannot communicate to the gateway in datapath
+ DPDK Compute cannot communicate to the gateway in datapath- cannot ping
+ the gateway
Revision history for this message
kratna (kratna) wrote :

The deployment /provisioning did not add

[VIRTUAL-HOST-INTERFACE]
physical_interface=bond0
name=vhost0
gateway=10.88.0.254
ip = 10.88.0.80/24 <-- this vhost0 IP statement was missing in the contrail-vrouter-agent.conf file
compute_node_address = 10.88.0.80
[SERVICE-INSTANCE]
netns_command=/usr/bin/opencontrail-vrouter-netns

shajuvk (shajuvk)
tags: added: sanityblocker
Revision history for this message
kratna (kratna) wrote :

Hari,

I have a setup in problem state

1. one compute in 4.1 - 10.88.0.81/24
2. second compute in 4.0.2 - 10.88.0.82/24

Internal network VIP = 10.88.0.20/24

The compute (4.0.2) is able to ping the VIP - 10.88.0.20 even when the IP statement is commented out in contrail-vrouter-agent.conf

whereas the compute node ( 4.1 ) is not able to ping the VIP - 10.88.0.20 when the IP statement is commented out in the contrail-vrouter-agent.conf

What we found is that when the

nh --get shows that for 4.1 ( when the IP statement is commented out) - type = drop
where as
for 4.0.2 - the type is resolve .

Please see Kiran's findings :

In the working case, nh for the local ip is receive nh.
But in the non working case, the nh is drop.
You may need to check with Hari why the behavior changed.

Non-working:
[root@overcloud-contraildpdk-1 ~]# rt --get 10.88.0.81/32 --vrf 0
Match 10.88.0.81/32 in vRouter inet4 table 0/0/unicast

Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP
vRouter inet4 routing table 0/0/unicast
Destination PPL Flags Label Nexthop Stitched MAC(Index)
0.0.0.0/0 0 T - 1 -
[root@overcloud-contraildpdk-1 ~]# nh --get 1
Id:1 Type:Drop Fmly: AF_INET Rid:0 Ref_cnt:1278 Vrf:0
              Flags:Valid, Etree Root,

Working:
[root@overcloud-contraildpdk-1 ~]# rt --get 10.88.0.81/32 --vrf 0
Match 10.88.0.81/32 in vRouter inet4 table 0/0/unicast

Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP
vRouter inet4 routing table 0/0/unicast
Destination PPL Flags Label Nexthop Stitched MAC(Index)
10.88.0.81/32 0 T - 9 -
[root@overcloud-contraildpdk-1 ~]# nh --get 9
Id:9 Type:Receive Fmly: AF_INET Rid:0 Ref_cnt:4 Vrf:1
              Flags:Valid, Policy(R), Etree Root,
              Oif:1

Revision history for this message
Vinod Nair (vinodnair) wrote :

AFAIK this is more of a provision issue.. Agent.conf should have the vhost ip

Revision history for this message
Michael Henkel (mhenkel-3) wrote : Re: [Bug 1751124] Re: DPDK Compute cannot communicate to the gateway in datapath- cannot ping the gateway
Download full text (8.9 KiB)

no;)

Regards,
Michael

> Am 02.03.2018 um 12:31 schrieb Vinod Nair <email address hidden>:
>
> AFAIK this is more of a provision issue.. Agent.conf should have the
> vhost ip
>
> --
> You received this bug notification because you are a bug assignee.
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.launchpad.net_bugs_1751124&d=DwIFaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=IIpdzrKFE-fFt447an76T47XLh_Zf5gZoVC_UG0ewoQ&m=C_KdVG2J0hJUm69MtFYfTFmEzWs75xiT4fX_dQQQVBk&s=zo-MX6gEu9LxcPaAuRagTtKLwWb4yutSr0O7l710114&e=
>
> Title:
> DPDK Compute cannot communicate to the gateway in datapath- cannot
> ping the gateway
>
> Status in Juniper Openstack:
> New
> Status in Juniper Openstack r4.1 series:
> New
> Status in Juniper Openstack trunk series:
> New
>
> Bug description:
> During RHOPS10 + Contrail 4.1 deployment ,
>
> The DPDK compute is unable to ping the gateway in datapath.
>
> ip a
>
> <none>
> [root@overcloud-contraildpdk-1 ~]# ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
> 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 0c:c4:7a:81:a4:a4 brd ff:ff:ff:ff:ff:ff
> inet 192.168.30.53/24 brd 192.168.30.255 scope global eno1
> valid_lft forever preferred_lft forever
> inet6 fe80::ec4:7aff:fe81:a4a4/64 scope link
> valid_lft forever preferred_lft forever
> 3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 0c:c4:7a:81:a4:a5 brd ff:ff:ff:ff:ff:ff
> inet6 fe80::ec4:7aff:fe81:a4a5/64 scope link
> valid_lft forever preferred_lft forever
> 4: enp2s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
> link/ether 90:e2:ba:da:fa:30 brd ff:ff:ff:ff:ff:ff
> 5: enp2s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
> link/ether 90:e2:ba:da:fa:31 brd ff:ff:ff:ff:ff:ff
> 7: ens7f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> link/ether 0c:c4:7a:b7:2c:fb brd ff:ff:ff:ff:ff:ff
> inet6 fe80::ec4:7aff:feb7:2cfb/64 scope link
> valid_lft forever preferred_lft forever
> 9: vhost0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
> link/ether 0c:c4:7a:b7:2c:fa brd ff:ff:ff:ff:ff:ff
> inet 10.88.0.81/24 brd 10.88.0.255 scope global vhost0
> valid_lft forever preferred_lft forever
> inet6 fe80::ec4:7aff:feb7:2cfa/64 scope link
> valid_lft forever preferred_lft forever
> 10: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
> link/ether c2:34:32:7d:ee:ce brd ff:ff:ff:ff:ff:ff
> 11: vlan213@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
> link/ether 0c:c4:7a:81:a4:a5 brd ff:ff:ff:ff:ff:ff
> inet 10.88.2.30/24 brd 10.88.2.255 scope global vlan213
> vali...

Read more...

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/40263
Submitter: Naveen N (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/40264
Submitter: Naveen N (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/40263
Committed: http://github.com/Juniper/contrail-controller/commit/e0848f3a2ee9d0f487cf0e5335527f25b16edb3d
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit e0848f3a2ee9d0f487cf0e5335527f25b16edb3d
Author: Naveen N <email address hidden>
Date: Sat Mar 3 13:21:16 2018 +0530

* Handle DHCP ip update on vhost interface

After changes to treat vhost as a VMI, DHCP IP update of vhost
was not handled. If IP was not set in configuration file, if address
updates comes primary ip address and subnet prefix length
Test case for same.

Change-Id: Ib4a6153536f25ae9ff1b8d681fd9270881fd1ff5
Closes-bug: #1751124

kratna (kratna)
information type: Proprietary → Public
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.