DPDK: Inter VM communication of iperf3 TCP throughput is very low on same host compare to non DPDK throughput

Hello. Just several questions about setup you are running.

1 - What is NUMA topology on your host?
2 - If there is NUMA topology, how PMD threads are spreading across NUMA nodes?

Best performance might be achieved if all PMD threads (as well as all vCPU pinned to VMs) are located on a same NUMA node.

For more detailed information about performance tuning please refer to https://github.com/openvswitch/ovs/blob/branch-2.6/INSTALL.DPDK-ADVANCED.md

Hello Matov,
 Thank you. Test PC is single NUMA only.I am not doing NIC binding as only trying to validate inter-VM communication in same host. PFB my PC configurations.

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Stepping: 2
CPU MHz: 1212.000
CPU max MHz: 2400.0000
CPU min MHz: 1200.0000
BogoMIPS: 4794.08
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 15360K
NUMA node0 CPU(s): 0-11
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1g b rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_t sc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline _timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ep t vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts

I am following INSTALL.DPDK.ADVANCED.md but no clue on low throughput.

Hello. Thanks for the information.

Let's try following scenario:
- Enable CPU pinning for Nova-compute, specify CPU list in nova.conf
- Isolate CPUs for nova in /etc/default/grub (reboot requires)
- Create flavor with 2 vCPUs and HugePages
- Spawn 2 VMs

Then you can use taskset command to run Iperf3 client/server on specific vCPU.
Also make sure that OVS pmd threads are not affected by nova vcpu pinning.

But generally speaking I can assume that this issue is not related to networking-ovs-dpdk. For more useful information you probably should take discussion in DPDK-users or OVS-discuss mailing lists.

Hello Matov,
Thanks for the suggestion, soon I will share the results of tcp iperf3 with taskset. The intention to post in networking-ovs-dpdk is to get inputs on all DPDK configurations in local.conf and to check for its correctness.

removing devstack as all dpdk logic for devstack is provided by the networking-ovs-dpdk plugin.

as far as i am aware the DPDK launchpad project is not used so this should be repoted to the dpdk mailing list.

networking-ovs-dpdk installs a wrapper file around qemu to workaround limitaition in old nova and qemu versions.

https://github.com/openstack/networking-ovs-dpdk/blob/d20da35a272e937802680ce8084d97c60ef629cc/devstack/libs/ovs-dpdk#L79-L128

with mitaka and qemu 2.3+ certenly 2.5+ there is no reason that i am aware of to continue to use this wrapper.

we keep it in the repo to support older software revisions.

one of the things that was broken with older version of qemu was the offload negotiation.
as such we forcable disable all offloads
VIRTIO_OPTIONS="csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off"
can you try removing
https://github.com/openstack/networking-ovs-dpdk/blob/d20da35a272e937802680ce8084d97c60ef629cc/devstack/libs/ovs-dpdk#L98-L103
and which will allow qemu to negotiate what offloads are available such as tso or check sum offloads.

The root cause of the issue is that currently ovs-dpdk does not support TSO offload. With the offload enable, the throughput will not lower than traditional path

Hello All,
Thanking you. Now I got clarity in the throughput difference. Yes, by default TOS is enabled in the test VM for without DPDK case and in DPDK TOS is disabled. so my test setup for comparing throughput with and without DPDK is not same.

Its TCP segment Offload TSO not TOS

Hi,

What is the current state about the ovs-dpdk TSO offload feature? Is this planned any time soon?

The only things that I could find so far are:
A Request For Comments was created by Intel and the TSO feature was showcased, but this feature is not yet implemented:
https://mail.openvswitch.org/pipermail/ovs-dev/2016-June/316414.html
https://ovs2016.sched.com/event/8aZf/optimizing-communications-grade-tcp-workloads-in-an-ovs-based-nfv-deployment-mark-kavanagh-intel
https://www.youtube.com/watch?v=hEP0_Bd3wrA

Thanks!