Ubuntu
qemu-kvm package

MTU not honored in virtio vnet

Bug #1297487 reported by Umberto Boscolo
36
This bug affects 5 people
Affects Status Importance Assigned to Milestone
QEMU
Won't Fix
Undecided
Unassigned
Salt
New
Undecided
Unassigned
qemu-kvm (Ubuntu)
Confirmed
High
Unassigned

Bug Description

I am observing a potential regression/different behavior between rel. 14.04 (dev branch) and release 13.04.

My hardware is: Cisco UCS blade B200-M3 and the network adapter card: Cisco UCS VIC 1240.

lsb_release -rd
Description: Ubuntu Trusty Tahr (development branch)
Release: 14.04

uname -a
Linux konan2 3.13.0-19-generic #40-Ubuntu SMP Mon Mar 24 02:36:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

The problem:
After starting a kvm with virtio interfaces I am passing HTTP traffic via an external network traffic simulator.
The tool is sending a TCP packet of 3481B, because the tool MTU is set to 1400B, it splits the packets into 3 TCP segments.
When the 3 segments are received at the host eth1 interface, the host (ubuntu 14.04) reassembles the TCP packets into a larger packet (GRO), then passes the packet up on vnet1. At this point, because vnet1 MTU is 1500B, it is supposed to re-segment the packet and pass the 3 segments up to the VM. But it passes the big 3481B packet instead.

This behavior did not happen when I tried the same scenario in release 13.04

I can disable this behavior by disabling TSO (TCP segment offloading in the vnet), but I did not have to do this in rel. 13.04 and I feel the MTU is not honored as it should be with rel. 14.04.

 ip link show | grep eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UP mode DEFAULT group default qlen 1000
 ip link show | grep vnet1
16: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1 state UNKNOWN mode DEFAULT group default qlen 500

I am attaching two tcpdump/pcap traces that show a TCP transaction passing on vnet1 when TSO is on and when TSO is off.

Please see:

- vnet1_tso_on.pcap
- vnet1_tso_off.pcap

in attachment.

I noticed there was a driver upgrade in rel. 14.04:

in 14.04:

ethtool -i eth1
driver: enic
version: 2.1.1.50
firmware-version: 2.1(3a)
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

ethtool -i vnet1
driver: tun
version: 1.6
firmware-version:
bus-info: tap
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

ethtool -k vnet1
Features for vnet1:
rx-checksumming: off [fixed]
tx-checksumming: on
 tx-checksum-ipv4: off [fixed]
 tx-checksum-ip-generic: on
 tx-checksum-ipv6: off [fixed]
 tx-checksum-fcoe-crc: off [fixed]
 tx-checksum-sctp: off [fixed]
scatter-gather: on
 tx-scatter-gather: on
 tx-scatter-gather-fraglist: on
tcp-segmentation-offload: off
 tx-tcp-segmentation: off
 tx-tcp-ecn-segmentation: off
 tx-tcp6-segmentation: off
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]

in 13.04 :

ethtool -i eth1
driver: enic
version: 2.1.1.39
firmware-version: 2.1(3a)
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

ethtool -i vnet1
driver: tun
version: 1.6
firmware-version:
bus-info: tap
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

ethtool -k vnet1
Features for vnet1:
rx-checksumming: off [fixed]
tx-checksumming: on
 tx-checksum-ipv4: off [fixed]
 tx-checksum-ip-generic: on
 tx-checksum-ipv6: off [fixed]
 tx-checksum-fcoe-crc: off [fixed]
 tx-checksum-sctp: off [fixed]
scatter-gather: on
 tx-scatter-gather: on
 tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
 tx-tcp-segmentation: on
 tx-tcp-ecn-segmentation: on
 tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]

Revision history for this message
Umberto Boscolo (uboscolo) wrote :
Revision history for this message
Umberto Boscolo (uboscolo) wrote :
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks for reporting this bug. I wonder if this may have been fixed since 1.7 - Are you able to reproduce this with the qemu version in ppa:ubuntu-virt/candidate? (That is the version which we hope will end up in 14.04 before release)

Changed in qemu-kvm (Ubuntu):
status: New → Incomplete
importance: Undecided → Medium
Revision history for this message
Umberto Boscolo (uboscolo) wrote :

I have updated to 1.7 but I still see the same issue.

 kvm --version
QEMU emulator version 1.7.0 (Debian 1.7.0+dfsg-3ubuntu7), Copyright (c) 2003-2008 Fabrice Bellard

drivers appears to be the same though

ethtool -i eth1
driver: enic
version: 2.1.1.50
firmware-version: 2.1(3a)
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

 ethtool -i vnet1
driver: tun
version: 1.6
firmware-version:
bus-info: tap
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

---

I can definitely see my VM receiving packets bigger than 1500B

tcpdump ....

....

Wednesday March 26 2014
INBOUND>>>>> 19:21:31:127
[ver:4 len:3481 proto:ether hop:0 t1:none vlan:0x010b] 802.1Q vlan#2067 P0 192.188.67.15.80 > 101.8.11.217.32103: P [bad tcp cksum 1df9!] 122145215:122148626(3411) ack 120888245 win 843 <nop,nop,timestamp 1001142818 1001142725> (DF) (ttl 64, id 61505, len 3463)

Wednesday March 26 2014
INBOUND>>>>> 19:21:31:127
[ver:4 len:2133 proto:ether hop:0 t1:none vlan:0x010b] 802.1Q vlan#2067 P0 192.188.67.17.80 > 101.8.15.65.23222: P [bad tcp cksum e6f3!] 112090738:112092801(2063) ack 107977615 win 843 <nop,nop,timestamp 1001142829 1001142718> (DF) (ttl 64, id 28330, len 2115)

....

ip link show dev vnet1
16: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UNKNOWN mode DEFAULT group default qlen 2500
    link/ether fe:54:00:f9:ea:e4 brd ff:ff:ff:ff:ff:ff

Serge Hallyn (serge-hallyn)
Changed in qemu-kvm (Ubuntu):
status: Incomplete → New
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks, but that looks like the version currently in 14.04 archive. Could you test with the 2.0~git-20140325.7b770c7-0ubuntu1 package which is in ppa:ubuntu-virt/candidate?

Changed in qemu-kvm (Ubuntu):
status: New → Incomplete
Revision history for this message
Umberto Boscolo (uboscolo) wrote :

I can see the following two installed, what am I missing?

sudo apt-cache policy qemu
qemu:
  Installed: 2.0~git-20140325.7b770c7-0ubuntu1
  Candidate: 2.0~git-20140325.7b770c7-0ubuntu1
  Version table:
 *** 2.0~git-20140325.7b770c7-0ubuntu1 0
        500 http://ppa.launchpad.net/ubuntu-virt/candidate/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     1.7.0+dfsg-3ubuntu7 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

sudo apt-cache policy qemu-kvm
qemu-kvm:
  Installed: 2.0~git-20140325.7b770c7-0ubuntu1
  Candidate: 2.0~git-20140325.7b770c7-0ubuntu1
  Version table:
 *** 2.0~git-20140325.7b770c7-0ubuntu1 0
        500 http://ppa.launchpad.net/ubuntu-virt/candidate/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     1.7.0+dfsg-3ubuntu7 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Ok, sorry, in comment #4 you had said you were on 1.7. There are some netdev commits between 1.7 and 2.0 so before marking this as affecting upstream I wanted to make sure you'd tested git head (which you effectively have).

Changed in qemu-kvm (Ubuntu):
status: Incomplete → New
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in qemu-kvm (Ubuntu):
status: New → Confirmed
Revision history for this message
Pierre Schweitzer (pierre-jean-schweitzer) wrote :

Also affected by this bug with Ubuntu 12.04LTS kvm.
A tracepath in a VM with virtio network card ends with: Resume: pmtu 65535.
While with pcnet or rtl8139 it properly ends with: Resume: pmtu 1500

This is pretty blocking given some servers are failing to communicate because of the wrong MTU (I've the case with a mail server).

Serge Hallyn (serge-hallyn)
Changed in qemu-kvm (Ubuntu):
importance: Medium → High
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

The bug description said this was a regression from 13.04 to 14.04, but comment #9 says 12.04 is affected.

Pierre, could you show us your kernel and qemu versions?

dpkg -l | egrep "(linux|qemu)"

Revision history for this message
Pierre Schweitzer (pierre-jean-schweitzer) wrote :
Download full text (8.0 KiB)

ii libselinux1 2.1.0-4.1ubuntu1 SELinux runtime shared libraries
ii linux-firmware 1.79.12 Firmware for Linux kernel drivers
ii linux-headers-3.2.0-60 3.2.0-60.91 Header files related to Linux kernel version 3.2.0
ii linux-headers-3.2.0-60-generic 3.2.0-60.91 Linux kernel headers for version 3.2.0 on 64 bit x86 SMP
ii linux-headers-3.2.0-61 3.2.0-61.92 Header files related to Linux kernel version 3.2.0
ii linux-headers-3.2.0-61-generic 3.2.0-61.92 Linux kernel headers for version 3.2.0 on 64 bit x86 SMP
ii linux-headers-server 3.2.0.61.72 Linux kernel headers on Server Equipment.
rc linux-image-2.6.32-21-server 2.6.32-21.32 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-22-server 2.6.32-22.36 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-24-server 2.6.32-24.43 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-25-server 2.6.32-25.45 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-26-server 2.6.32-26.48 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-27-server 2.6.32-27.49 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-29-server 2.6.32-29.58 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-30-server 2.6.32-30.59 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-33-server 2.6.32-33.72 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-34-server 2.6.32-34.77 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-35-server 2.6.32-35.78 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-37-server 2.6.32-37.81 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-38-server 2.6.32-38.83 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-39-server 2.6.32-39.86 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-40-server 2.6.32-40.87 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-2.6.32-41-server 2.6.32-41.88 Linux kernel image for version 2.6.32 on x86_64
rc linux-image-3.2.0-24-generic 3.2.0-24.39 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
rc linux-image-3.2.0-25-generic 3.2.0-25.40 Linux kernel image for version 3.2.0 on 64 bit x86 SMP
rc linux-image-3.2.0-26-generic 3.2.0-26.41 Linux kernel image for version 3.2.0 on 64 bit ...

Read more...

Revision history for this message
mp (mp-xmission) wrote :

I believe this bug may be breaking Salt by breaking ZeroMQ in certain virtualized environments. Additional information here: https://github.com/saltstack/salt/issues/12248

Revision history for this message
Ian Wells (ijw-ubuntu) wrote :

Seems to me that the issue is not that the MTU isn't being honoured, but that the MTU should be checked *before* TSO assembly and not *after*. Assembly should happen outside the VM if the VM has enabled it on the interface internally, and clearly the incoming segments (not packets, at this point) may be longer than the MTU.

Revision history for this message
Thomas Huth (th-huth) wrote :

Did anybody ever tried to reproduce this bug with upstream QEMU?

Changed in qemu:
status: New → Incomplete
Revision history for this message
Thomas Huth (th-huth) wrote :

Closing for QEMU since there hasn't been any response within a year.

Changed in qemu:
status: Incomplete → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.