Vm-2-Vm over OVS iperf limited to 2.5 Gb

Bug #1403047 reported by Alexander Fridman
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Egor Kotko
6.1.x
Fix Released
High
Egor Kotko

Bug Description

Fuel Setup :
  * Fuel 6.0 build 48.
  * Deployment : CentOS, Neutron with VLAN segmentation
  * Installed with Mellanox NIC with driver-only mode (non-SRIOV), OVS mechanism used.

Test setup :
  * 2 VM on compute-1
  * 1 VM on compute-2

Test :
  * Run server: iperf –s
  * On other VMs run iperf –c $server_IP –t 30 –P 2 –I 1 –f -g

Results:
  * VM-2-VM on the same compute : 8.7 Gb
  * VM-2-VM on the different computes : 1.5 Gb
  * 2 VMS on compute-1 to VM on compute 2 : 2.2Gb (1.3Gb + 0.9Gb)

==================================================================
On Fuel 5.1 I see the same problem and I see the problem came from OVS RX side.
Fuel Setup :
  * Fuel 5.1 final
  * Deployment : CentOS, Neutron with VLAN segmentation
  * Installed with Mellanox NIC with driver-only mode (non-SRIOV), OVS mechanism used.
  * Network/NIC speed is 10Gb

Results :
  * Performance from VM to physical server is : 5.8Gb
  * While Performance from physical server to VM is only : 1.8-2.5 Gb (This indicates that problem is on VM RX side)

==================================================================
The issue does no occur with SRIOV deployment.
 * On same HW setup with Mellanox driver + SRIOV over 56Gb network VM-2-VM performance is 40 Gb.

==================================================================
API-Version for Fuel 6.0 b48 :
{"build_id": "2014-12-09_19-59-53", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "48", "auth_required": true, "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-09_19-59-53", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "48", "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2f7634f8c0132f267a8ce390477784ce56fc18d8"}}}, "fuellib_sha": "2f7634f8c0132f267a8ce390477784ce56fc18d8"}

==================================================================
API-Version for Fuel 5.1 Final :
{"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}

Revision history for this message
Alexander Fridman (t-sasha) wrote :
Changed in fuel:
milestone: none → 6.0
importance: Undecided → High
Changed in fuel:
assignee: nobody → Fuel Partner Integration Team (fuel-partner)
Changed in fuel:
assignee: Fuel Partner Integration Team (fuel-partner) → MOS Linux (mos-linux)
Changed in fuel:
milestone: 6.0 → 6.1
Changed in fuel:
status: New → Confirmed
Mike Scherbakov (mihgen)
tags: added: release-notes
Changed in fuel:
status: Confirmed → Won't Fix
no longer affects: fuel/6.0.x
Revision history for this message
Tomasz 'Zen' Napierala (tzn) wrote :

Please provide some details/reasons why it's won't fix

Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

It won't be fixed in 6.0, because 2.5Gb is not something bad, but rather a good result.

For example, results here show only 1.5Gb:
https://wiki.openstack.org/wiki/Ovs-flow-logic

Recent presentations on OVS performance show about 260K pps (about 3 Gbps with 1500-byte packets):
http://www.slideshare.net/andyhky/open-v-switchfinal

There are various way to improve OVS performance, and we are investigating them, but it is not something that can be fixed for all environments.

On the other hand, 6.1 status is still 'Confirmed', because we may update OVS itself or update its configuration if tests show significant improvements.

tags: added: mos-linux
no longer affects: fuel/6.0.x
Revision history for this message
OSCI Robot (oscirobot) wrote :

package openvswitch has been built for project packages/centos6/openvswitch
Package version == 2.3.1, package release == 1

Changeset: https://review.fuel-infra.org/2782
project: packages/centos6/openvswitch
branch: 6.1
author: Aleksandr Mogylchenko
committer: Aleksandr Mogylchenko
subject: OpenVSwitch 2.3.1 LTS for CentOS
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-2782/

Revision history for this message
OSCI Robot (oscirobot) wrote :

package openvswitch has been built for project packages/precise/openvswitch
Package version == 2.3.1, package release == ubuntu1

Changeset: https://review.fuel-infra.org/2162
project: packages/precise/openvswitch
branch: 6.1
author: Aleksandr Mogylchenko
committer: Aleksandr Mogylchenko
subject: OpenVSwtich 2.3.1 (LTS branch) from Ubuntu
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /ubuntu-fuel-6.1-stable-2162/

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package openvswitch has been built for project packages/trusty/openvswitch
Package version == 2.3.1, package release == ubuntu1

Changeset: https://review.fuel-infra.org/4433
project: packages/trusty/openvswitch
branch: 6.1
author: Aleksandr Mogylchenko
committer: Aleksandr Mogylchenko
subject: OpenVswitch 2.3.1 from Vivid imported to MOS 6.1
status: patchset-created

Files placed on repository:
openvswitch-common_2.3.1-0ubuntu1~mos6.1_amd64.deb
openvswitch-dbg_2.3.1-0ubuntu1~mos6.1_amd64.deb
openvswitch-ipsec_2.3.1-0ubuntu1~mos6.1_amd64.deb
openvswitch-pki_2.3.1-0ubuntu1~mos6.1_all.deb
openvswitch-switch_2.3.1-0ubuntu1~mos6.1_amd64.deb
openvswitch-test_2.3.1-0ubuntu1~mos6.1_all.deb
openvswitch-testcontroller_2.3.1-0ubuntu1~mos6.1_amd64.deb
openvswitch-vtep_2.3.1-0ubuntu1~mos6.1_amd64.deb
python-openvswitch_2.3.1-0ubuntu1~mos6.1_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://obs-1.mirantis.com:82/trusty-fuel-6.1-stable-4433/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

package openvswitch has been built for project packages/trusty/openvswitch
Package version == 2.3.1, package release == ubuntu1

Changeset: https://review.fuel-infra.org/4433
project: packages/trusty/openvswitch
branch: 6.1
author: Aleksandr Mogylchenko
committer: Alexei Sheplyakov
subject: Import OpenVswitch 2.3.1 to MOS 6.1
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /trusty-fuel-6.1-stable-4433/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/167551

Changed in fuel:
assignee: Aleksander Mogylchenko (amogylchenko) → Stanislav Makar (smakar)
Revision history for this message
OSCI Robot (oscirobot) wrote :

package openvswitch has been built for project packages/trusty/openvswitch
Package version == 2.3.1, package release == ubuntu1

Changeset: https://review.fuel-infra.org/4433
project: packages/trusty/openvswitch
branch: 6.1
author: Aleksandr Mogylchenko
committer: Alexei Sheplyakov
subject: Import OpenVswitch 2.3.1 to MOS 6.1
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /trusty-fuel-6.1-stable/

Revision history for this message
OSCI Robot (oscirobot) wrote :

package openvswitch has been built for project packages/centos6/openvswitch
Package version == 2.3.1, package release == 1

Changeset: https://review.fuel-infra.org/2782
project: packages/centos6/openvswitch
branch: 6.1
author: Aleksandr Mogylchenko
committer: Aleksandr Mogylchenko
subject: OpenVSwitch 2.3.1 LTS for CentOS
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /centos-fuel-6.1-stable/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/167551
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=442c82e3fc3d7c9c50b7188f204c8f283c8dbcda
Submitter: Jenkins
Branch: master

commit 442c82e3fc3d7c9c50b7188f204c8f283c8dbcda
Author: Stanislav Makar <email address hidden>
Date: Wed Mar 25 10:21:19 2015 +0000

    Install openvswitch-datapath-dkms package

    * Install openvswitch-datapath-dkms package in any way for Ubuntu 14.04
    due to upgrading to OpenvSwitch 2.3
    * Clean up unnecessary kern_has_ovs_datapath fact

    Change-Id: I759b8507bdbc98b1b9931ca0ac2a066dfb7b3bb4
    Partial-Bug: #1403047
    Docimpact: OpenvSwitch 2.3 support

Revision history for this message
Stanislav Makar (smakar) wrote :

now it is needed to be tested after OVS 2.3 merge

Revision history for this message
Egor Kotko (ykotko) wrote :

Ubuntu, int: 2x10G - bond(balace-rr)
Results:
  * VM-2-VM on the same compute : 28.1 Gbits/sec
http://paste.openstack.org/show/198154/
 * VM-2-VM on the different computes : 3.93 Gbits/sec
http://paste.openstack.org/show/198160/
  * 2 VMS on compute-1 to VM on compute 2 : 5.36Gb (2.85Gb + 2.51Gb)
http://paste.openstack.org/show/198162/

Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

Verified.

tags: added: release-notes-done
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-docs (stable/6.1)

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/194961

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-docs (stable/6.1)
Download full text (45.4 KiB)

Reviewed: https://review.openstack.org/194961
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=0e26e7d7cc153d179ec34985645dd23cdd239ddb
Submitter: Jenkins
Branch: stable/6.1

commit 5cc5f0c643aebecaf3bf4580535a3ea7c3334a6c
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:43:35 2015 -0700

    Removed streamlined patching backend pieces

    Change-Id: I955e76ccdbd12a9145f4e9b689f80bdf9fcaf929

commit 563c4b5c78ebfcb1f4f91047c2919f6270f9a1d4
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:30:30 2015 -0700

    Removed outdated patching guide

    Change-Id: I76180c277789ade9c5ebedd19fe2092847c0b7d9

commit 8d120c14bec1ab41d448683ad146a3053a57c4ee
Author: Irina Povolotskaya <email address hidden>
Date: Tue Jun 23 19:59:11 2015 +0300

    Add dual hypervisor ref arch into 6.1 docs

    Change-Id: I900c24c9de878eafadbfc995aa879b7f55737fac

commit feebd1592d3305b64bbdfd0bc5fe108190aef120
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:38:17 2015 +0300

    [OPs guide] Running Ceilometer section edits

    1. conf file extract is updated
    2. note is updated

    Closes-bug: 1467817
    Change-Id: I0217e164108e0ba6c1397045a5e57d13ff429223

commit 44a93f9dead7511a3461ec35248dbb689c81eafd
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:04:40 2015 +0300

    [RN6_1] Final changes

    1. capitalization
    2. 2014.2 to 2014.2.2
    3. general improvements

    Change-Id: I45057e90c90550559f66bc67ccdf97a559fd9000

commit bb41389cae58084285688853281516b659686422
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 16:45:35 2015 +0300

    Update patching decription

    Update patching description with
    the standard Linux commands.

    Change-Id: Ia1a8346639c468fdfce15a11d2430bf3a4731244

commit bf3018fae3f2e564413d33aba6cdebf8868f0b4e
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 15:55:49 2015 +0300

    [RN6_1] Clean up

    1. Rearranges sections
    2. Improves RST
    3. Changes titles order

    Change-Id: I6110bf515667d3d6ba08ad35ff5d593dbc96641e

commit 1c7e4457808e8f2d6c56fdf31252170972e444b9
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 15:26:28 2015 +0300

    Replaces VBOX screenshots

    This patch:
    - replaces VBOX screenshots
    - changes the link for Download Mirantis VirtualBox scripts
     to https://docs.mirantis.com/openstack/fuel/fuel-master/#downloads

    Change-Id: I58dede960c5c3355d39b07ff44b757403f6af02c
    Closes-Bug: #1467872

commit 0a568bf53fc0e25d1d692d5d74b4a7b4d983bbcc
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 14:01:55 2015 +0300

    6.1 --separate repos

    change wording and add links to the
    separate repos feature.

    Change-Id: Ib5d0778a0d8f1534f79ed2f553574cb69a3150b0

commit 95a188b21cbdd064d92696b7920e6a0105fe0c56
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 12:07:28 2015 +0300

    Corrects the output 'pcs status'

    Changes the example outputs to appropriate ones.

    Change-Id: Ib6d83...

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on packages/precise/openvswitch (6.1)

Change abandoned by Aleksandr Mogylchenko <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/2162

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.