Ubuntu
qemu-kvm package

tap interface drops many packets on highload systems

Bug #1423631 reported by Michael Kazakov on 2015-02-19

This bug affects 7 people

	Status	Importance	Assigned to
linux (Ubuntu)	Confirmed	Medium	Unassigned
qemu (Ubuntu)	Incomplete	Medium	Unassigned
qemu-kvm (Ubuntu)	Incomplete	Medium	Unassigned

Bug Description

I use qemu-kvm in openstack. On highload hypervisor tap interface of net-highload guest drops many TX packets.
Network options of qemu-system-x86_64 "... -netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:86:67:7b,bus=pci.0,addr=0x3..."

Network domin config:

tape4009073-0b Link encap:Ethernet HWaddr fe:16:3e:86:67:7b
          inet6 addr: fe80::fc16:3eff:fe86:677b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:1587622634 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1484106438 errors:0 dropped:460259 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:877878711500 (877.8 GB) TX bytes:3071846828531 (3.0 TB)
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Feb 18 12:06 seq
crw-rw---- 1 root audio 116, 33 Feb 18 12:06 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
InstallationDate: Installed on 2014-07-01 (234 days ago)
InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2)
MachineType: Dell Inc. PowerEdge M620
Package: qemu-kvm
PciMultimedia:

ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
LC_MESSAGES=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.16.0-30-generic root=UUID=5965239d-820a-49a1-9d8e-23a4c1f87ce6 ro
ProcVersionSignature: Ubuntu 3.16.0-30.40~14.04.1-generic 3.16.7-ckt3
RelatedPackageVersions:
linux-restricted-modules-3.16.0-30-generic N/A
linux-backports-modules-3.16.0-30-generic N/A
linux-firmware 1.127.11
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.16.0-30-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 01/21/2014
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.2.7
dmi.board.name: 0T36VK
dmi.board.vendor: Dell Inc.
dmi.board.version: A01
dmi.chassis.type: 25
dmi.chassis.vendor: Dell Inc.
dmi.chassis.version: PowerEdge M1000e
dmi.modalias: dmi:bvnDellInc.:bvr2.2.7:bd01/21/2014:svnDellInc.:pnPowerEdgeM620:pvr:rvnDellInc.:rn0T36VK:rvrA01:cvnDellInc.:ct25:cvrPowerEdgeM1000e:
dmi.product.name: PowerEdge M620
dmi.sys.vendor: Dell Inc.

See original description

Tags:

Michael Kazakov (gnomino) on 2015-02-19

summary:	- tap interface drors may packages on highload sysstems + tap interface drops many packages on highload systems
summary:	- tap interface drops many packages on highload systems + tap interface drops many packets on highload systems

Revision history for this message

Serge Hallyn (serge-hallyn) wrote on 2015-02-20:

Please update the bug with system information by doing

apport-collect 1423631

Revision history for this message

Brad Figg (brad-figg) wrote on 2015-02-20: Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1423631

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status:	New → Incomplete

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: BootDmesg.txt

BootDmesg.txt Edit (143.7 KiB, text/plain)

apport information

tags:	added: apport-collected trusty
description:	updated

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: CurrentDmesg.txt

CurrentDmesg.txt Edit (91.3 KiB, text/plain)

apport information

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: IwConfig.txt

IwConfig.txt Edit (3.1 KiB, text/plain)

apport information

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: Lspci.txt

Lspci.txt Edit (130.2 KiB, text/plain)

apport information

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: Lsusb.txt

Lsusb.txt Edit (417 bytes, text/plain)

apport information

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: ProcCpuinfo.txt

ProcCpuinfo.txt Edit (45.6 KiB, text/plain)

apport information

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: ProcInterrupts.txt

ProcInterrupts.txt Edit (29.7 KiB, text/plain)

apport information

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: ProcModules.txt

#10

ProcModules.txt Edit (4.7 KiB, text/plain)

apport information

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: UdevDb.txt

#11

UdevDb.txt Edit (650.9 KiB, text/plain)

apport information

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: UdevLog.txt

#12

UdevLog.txt Edit (1.2 MiB, text/plain)

apport information

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-02-20: WifiSyslog.txt

#13

WifiSyslog.txt Edit (1.1 KiB, text/plain)

apport information

Changed in qemu-kvm (Ubuntu):
status:	New → Incomplete
status:	Incomplete → Confirmed

Serge Hallyn (serge-hallyn) on 2015-02-25

Changed in linux (Ubuntu):
status:	Incomplete → New

Revision history for this message

Brad Figg (brad-figg) wrote on 2015-02-25: Status changed to Confirmed

#14

This change was made by a bot.

Changed in linux (Ubuntu):
status:	New → Confirmed

Chris J Arges (arges) on 2015-06-17

description:

updated

Revision history for this message

Chris J Arges (arges) wrote on 2015-06-17:

#15

Can you explain more about your setup?
Which version of openstack are you running?
How have you setup your instance?
Which operating system / kernel version are you running in your instance?
How are you generating outbound traffic from the instance?
Is the outbound traffic going through other equipment which may be dropping those packets?

In my Trusty/Icehouse deploy, I've deployed instanced and observed the tapXXXXX interfaces on the compute node and noticed no large amount of dropped TX packets.

Changed in qemu-kvm (Ubuntu):
status:	Confirmed → Incomplete
importance:	Undecided → Medium
Changed in qemu (Ubuntu):
importance:	Undecided → Medium
status:	New → Incomplete
Changed in linux (Ubuntu):
importance:	Undecided → Medium

Revision history for this message

Michael Kazakov (gnomino) wrote on 2015-06-18:

#16

I found the cause of the problem.
The tap interface of a instance has small tx length 500 packets. At a high network load (15-25 Kpps) and high CPU load on a guest some times queue overflows and the tap interface start drop network packages. I think that queue overflow occurs in moments of micro friezes (<1000ms) of a virtual machine. I solved this problem by increasing the tx_queue up to 10000.

Revision history for this message

sean redmond (sean-redmond1) wrote on 2015-10-12:

#17

I also see this bu only on Windows guests.

Revision history for this message

sean redmond (sean-redmond1) wrote on 2015-10-12:

#18

I was able to resolve this with the below udev rule:

# cat /etc/udev/rules.d/60-tap.rules
KERNEL=="tap*", RUN+="/sbin/ip link set %k txqueuelen 10000"
#

I used the below to reload udev

udevadm control --reload-rules

I then used the below to apply the rules to already created interfaces:

udevadm trigger --attr-match=subsystem=net

Revision history for this message

Satish Patel (satish-txt) wrote on 2018-09-16:

#19

We are having same issue on CentOS7 and i have increase tx_queue upto 10000 but i am still seeing TX drops on tap interface.

Question:

Can i increase tx_queue on interface with "ifconfig <tap_interface> txqueue 10000" without rebooting machine?

I have to do it at boot time?

Revision history for this message

Satish Patel (satish-txt) wrote on 2019-07-09:

#20

This is fact, you can't get high PPS with tap interface because they run in kernel space and it drive packets on kernel which isn't scalable.

Finally i have migrate all my compute nodes to SR-IOV and much happy now.

Brad Figg (brad-figg) on 2019-07-24

tags:

added: ubuntu-certified

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntuqemu-kvm package

tap interface drops many packets on highload systems

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
qemu-kvm package