Fuel for OpenStack

[X(L)710] MTU for private network on DPDK-enabled interface is configured incorrectly

  1. Series ocata
  2. Bug #1587310
Bug #1587310 reported by Mikhail Chernik
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
In Progress
High
Fedor Zhadaev
Fuel for OpenStack 11.0
Mitaka
Fix Released
High
MOS Maintenance
Fuel for OpenStack 9.2-mu-2
Newton
In Progress
High
Fedor Zhadaev
Fuel for OpenStack 10.1
Ocata
In Progress
High
Fedor Zhadaev
Fuel for OpenStack 11.0

Bug Description

Environment:
MOS 9.0, ISO 400, NIC: XL710, dpdk driver igb_uio.

Detailed description

Due to wrong MTU setting network is unstable if a VM on DPDK private network is booted on image other than Cirros. MTU does not take into account 4 bytes VLAN tag.

#ps aux | grep dnsmasq | grep -o -P 'option:mtu[^ ]+'
option:mtu,1500

Steps to reproduce:
1. Deploy an env with Private network placed on a DPDK-enabled interface on a compute node
2. Create Ubuntu Cloud image
3. Set hw:mem_page_size=2048 metadata option for m1.smal flavor
4. Boot a VM from Ubuntu Cloud image on the compute with DPDK-enabled interface
5. When the VM is fully booted, ping it from controller and try to log into it via ssh

Expected result:
Both ping and ssh works

Actual result:
Ping works well, but ssh connection is closed after timeout.

Workaround:
On controller:
NS=$(ip netns list | grep qdhcp)
ip netns exec $NS ifconfig
DEV=$(ifconfig | grep tap | awk '{print $1}')
ip netns exec $NS i plink set mtu 1496 dev $DEV

On VM:
ip link set mtu 1496 dev eth0

See original description
Aleksey Kasatkin (alekseyk-ru)
tags: added: area-library
Revision history for this message
Mikhail Chernik (mchernik) wrote :

Environment: MOS 9.0, ISO 400

description: updated
Revision history for this message
Atsuko Ito (yottatsa) wrote :

MTU doesn't needed to be adjusted for 802.1Q AFAIK

Revision history for this message
Atsuko Ito (yottatsa) wrote :

Looks like dpdk driver and/or NIC specifics:
http://dpdk.org/ml/archives/dev/2015-August/023089.html
http://dpdk.org/dev/patchwork/patch/12918/

tags: added: area-linux
removed: area-library
tags: added: team-network
Revision history for this message
Sergey Matov (smatov) wrote :

According to neutron dhcp/ml2 configuration we need to check out 1 parameter for MTU.

In file
/etc/neutron/plugins/ml2/ml2_conf.ini
Parameter
physical_network_mtus

Revision history for this message
Atsuko Ito (yottatsa) wrote :

@smatov it should be noted as a workaround.

Also, it could be done by setting up MTU 1496 on Private interfaces on controllers and computes to apply this workaround out of the box.

Atsuko Ito (yottatsa)
summary: - MTU for private network on DPDK-enabled interface is configured
- incorrectly
+ [X(L)710] MTU for private network on DPDK-enabled interface is
+ configured incorrectly
Ilya Kutukov (ikutukov)
Changed in fuel:
importance: Undecided → High
status: New → Confirmed
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
Maksim Malchuk (mmalchuk)
no longer affects: fuel/newton
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Networking (l23-network)
Revision history for this message
Mikhail Chernik (mchernik) wrote :

Request for adding this issue to release notes: https://bugs.launchpad.net/fuel/+bug/1587867
Closed for 9.0.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to mos/mos-docs (master)

Related fix proposed to branch: master
Change author: Olena Logvinova <email address hidden>
Review: https://review.fuel-infra.org/22403

Olena Logvinova (ologvinova)
tags: added: release-notes
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Registry Administrators (registry) → l23network (l23network)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to mos/mos-docs (master)

Reviewed: https://review.fuel-infra.org/22403
Submitter: Evgeny Konstantinov <email address hidden>
Branch: master

Commit: 9bfce94c4aec46c05e744432fd307a020d4c62c7
Author: Olena Logvinova <email address hidden>
Date: Wed Jun 22 13:20:42 2016

[RN 9.0] [NFV] [Intel XL710 40] known issues

This patch adds the following bugs and their workarounds
to the RN 9.0 Known issues section:

https://bugs.launchpad.net/fuel/+bug/1583077
https://bugs.launchpad.net/fuel/+bug/1587310

Change-Id: Ie15d0888afd46599a0ed421bd757bf1d471504a5
Related-Bug: #1583077
Related-Bug: #1587310
Closes-Bug: #1587867

Olena Logvinova (ologvinova)
tags: added: release-notes-done
removed: release-notes
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to packages/trusty/dpdk (9.0)

Fix proposed to branch: 9.0
Change author: Sergey Matov <email address hidden>
Review: https://review.fuel-infra.org/28628

Revision history for this message
Sergey Matov (smatov) wrote :

Issue can be fixed with OVS 2.6.1 and DPDK 16.07 by configuring L2 MTU on dpdkX physical interface.
E.g. if we are setting MTU as default (1500 for layer 3) next command required for port named dpdk0

ovs-vsctl set Interface dpdk0 mtu_request=1518

Where 1518 is l3 mtu + eth header size + vlan tag size.

Appropriate changes will be applied for MOS 9.2

Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Registry Administrators (registry) → Fuel Sustaining (fuel-sustaining-team)
Vladimir Kuklin (vkuklin)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Sergey Matov (smatov)
Sergey Matov (smatov)
Changed in fuel:
status: Confirmed → In Progress
Dmitry Klenov (dklenov)
Changed in fuel:
assignee: Sergey Matov (smatov) → Mikhail Zhnichkov (mzhnichkov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/420350

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/420914

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Fedor Zhadaev (fzhadaev)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Fedor Zhadaev (fzhadaev) → Mikhail Zhnichkov (mzhnichkov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Fedor Zhadaev (fzhadaev)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/422724

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Fedor Zhadaev (fzhadaev) → Mikhail Zhnichkov (mzhnichkov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Fedor Zhadaev (fzhadaev)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/423629

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/423631

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/423629
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3bdad751d0c46ed964f1dcacffb701d59d7c9958
Submitter: Jenkins
Branch: stable/mitaka

commit 3bdad751d0c46ed964f1dcacffb701d59d7c9958
Author: Mikhail <email address hidden>
Date: Sat Jan 14 19:09:20 2017 +0400

    Correct MTU

    Correct MTU for interface with 'i40e' driver
    Closes-Bug: #1587310

    Change-Id: I2cee98194c4d1761feeb1999804c0e24a467eb2c

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/422724
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=747e897ae0238251066f7bd3f20c45ad49f7b20a
Submitter: Jenkins
Branch: stable/mitaka

commit 747e897ae0238251066f7bd3f20c45ad49f7b20a
Author: Mikhail <email address hidden>
Date: Mon Jan 16 23:40:10 2017 +0400

    Serialize mtu for dpdk interface with 'i40e' driver

    On NIC: XL710 with driver 'i40e' MTU does not take into
    account 4 bytes VLAN tag, so we should increase it manually

    DocImpact
    Change-Id: I3d95db9ec6fae4d8cd397c429d785dbdf1502b21
    Partial-Bug: #1587310
    Co-Authored-By: Fedor Zhadaev <email address hidden>
    (cherry picked from commit 737556d)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/423631
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4a64028d70d492afc0bc631f120e2a54ef076d48
Submitter: Jenkins
Branch: stable/newton

commit 4a64028d70d492afc0bc631f120e2a54ef076d48
Author: Mikhail <email address hidden>
Date: Sat Jan 14 19:09:20 2017 +0400

    Correct MTU

    Correct MTU for interface with 'i40e' driver
    Closes-Bug: #1587310

    Change-Id: I2cee98194c4d1761feeb1999804c0e24a467eb2c

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/420350
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4e8c035dc4c177f644171a1ea823fdaf14763992
Submitter: Jenkins
Branch: master

commit 4e8c035dc4c177f644171a1ea823fdaf14763992
Author: Mikhail <email address hidden>
Date: Sat Jan 14 19:09:20 2017 +0400

    Correct MTU

    Correct MTU for interface with 'i40e' driver
    Closes-Bug: #1587310

    Change-Id: I2cee98194c4d1761feeb1999804c0e24a467eb2c

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/424031

Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Reopening for master and newton because not all patches landed in these branches: https://review.openstack.org/#/q/topic:bug/1587310

Revision history for this message
Mikhail Chernik (mchernik) wrote :

Deploy failed on MOS9.2 RC2 (snapshot #822) with message:

Parameter mtu failed on L23_stored_config[ens11f1]: Validate method failed for class mtu: undefined method `to_i' for {"value"=>{"value"=>"1504"}}:Hash at /etc/puppet/modules/l23network/manifests/l2/port.pp:119

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/427773

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to mos/mos-docs (master)

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30384

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on mos/mos-docs (master)

Change abandoned by Evgeny Konstantinov <email address hidden> on branch: master
Review: https://review.fuel-infra.org/30384
Reason: will do a new commit

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/428214

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to mos/mos-docs (master)

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30442

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/428615

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/428616

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to mos/mos-docs (master)

Reviewed: https://review.fuel-infra.org/30442
Submitter: Olena Logvinova <email address hidden>
Branch: master

Commit: ec2096af34a329ece25984db03c0e6c4b614af1e
Author: Evgeny Konstantinov <email address hidden>
Date: Fri Feb 3 15:44:26 2017

Add i40e deployment failure issue to 9.2 relnotes

Change-Id: I6f2b6dc32b993ad899706db2417121472f096eed
Related-Bug: #1587310

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to mos/mos-docs (stable/9.2)

Related fix proposed to branch: stable/9.2
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/30468

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to mos/mos-docs (stable/9.2)

Reviewed: https://review.fuel-infra.org/30468
Submitter: Mariia Zlatkova <email address hidden>
Branch: stable/9.2

Commit: a60eafdc71ff08ba3633f68a266694ed7e09998c
Author: Evgeny Konstantinov <email address hidden>
Date: Fri Feb 3 15:54:09 2017

Add i40e deployment failure issue to 9.2 relnotes

Change-Id: I6f2b6dc32b993ad899706db2417121472f096eed
Related-Bug: #1587310
(cherry picked from commit ec2096af34a329ece25984db03c0e6c4b614af1e)

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Georgy Kibardin (gkibardin)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Georgy Kibardin (gkibardin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 11.0.0.0rc1

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Mikhail Zhnichkov (mzhnichkov) → Fedor Zhadaev (fzhadaev)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/420914
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=9c51c8e22ba7e69bfd84104d1c65a8a6df9acedc
Submitter: Jenkins
Branch: master

commit 9c51c8e22ba7e69bfd84104d1c65a8a6df9acedc
Author: Mikhail <email address hidden>
Date: Mon Jan 16 23:40:10 2017 +0400

    Serialize mtu for dpdk interface with 'i40e' driver

    On NIC: XL710 with driver 'i40e' MTU does not take into
    account 4 bytes VLAN tag, so we should increase it manually

    DocImpact
    Change-Id: I3d95db9ec6fae4d8cd397c429d785dbdf1502b21
    Partial-Bug: #1587310
    Co-Authored-By: Fedor Zhadaev <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/447955

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/newton)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/428615
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/mitaka)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/427773
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/428616
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f199c0ba29de6364c67140bfa48ff11c4bb33ded
Submitter: Jenkins
Branch: stable/mitaka

commit f199c0ba29de6364c67140bfa48ff11c4bb33ded
Author: Mikhail <email address hidden>
Date: Thu Feb 2 19:50:29 2017 +0400

    Fix vsctl call

    Change-Id: I700ff89d0ff90fabc1843f0e740ab38d7fbaa45a
    Closes-Bug: #1587310

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Can't reproduce bug on our test environment due to lack of appropriate HW. In accordance with our policy, fix should be verified on the customer side.

Revision history for this message
Daniel Forsberg (daniel-forsberg1) wrote :

Hi!

I can confirm that this still is a issue in mitaka-9.2.

Fix from here: https://review.openstack.org/#/c/427773/ is not applied

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on packages/trusty/dpdk (9.0)

Change abandoned by Sergey Matov <email address hidden> on branch: 9.0
Review: https://review.fuel-infra.org/28628

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/428214
Reason: This repo is retired now, no further work will get merged.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/ocata)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/ocata
Review: https://review.opendev.org/447955
Reason: This repo is retired now, no further work will get merged.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/newton)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/newton
Review: https://review.opendev.org/424031
Reason: This repo is retired now, no further work will get merged.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.