| 2022-08-05 08:41:49 |
Trent Lloyd |
bug |
|
|
added bug |
| 2022-08-05 08:44:03 |
Trent Lloyd |
linux (Ubuntu): assignee |
|
Trent Lloyd (lathiat) |
|
| 2022-08-05 08:44:05 |
Trent Lloyd |
linux (Ubuntu): status |
New |
Confirmed |
|
| 2022-08-05 08:44:10 |
Trent Lloyd |
tags |
|
sts |
|
| 2022-08-05 10:00:37 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2022-08-08 04:51:44 |
DUFOUR Olivier |
bug |
|
|
added subscriber DUFOUR Olivier |
| 2022-08-18 20:50:13 |
Jeff Lane |
bug |
|
|
added subscriber Intel Team |
| 2022-08-31 22:50:52 |
Jeff Lane |
linux (Ubuntu): status |
Confirmed |
Incomplete |
|
| 2022-08-31 23:00:02 |
Jeff Lane |
tags |
sts |
servcert-472 sts |
|
| 2022-09-16 06:29:25 |
Arif Ali |
bug |
|
|
added subscriber Arif Ali |
| 2022-09-30 16:26:19 |
Trent Lloyd |
description |
Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration.
Environment: Ubuntu OpenStack Focal-Ussuri with OVN
Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE
Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy
Host Error Messages:
ice 0000:98:00.1: VF 7 failed opcode 6, retval: -5
Guest Error Messages:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES
It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count.
There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue.
These Virtual Machines do work with the Mainline v5.19 build on the host and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related commits that are not currently backported to v5.15 or any upstream stable kernel:
6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18]
be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19]
Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug:
efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19]
It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. |
Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration.
Environment: Ubuntu OpenStack Focal-Ussuri with OVN
Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE
Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy
Host Error Messages:
ice 0000:98:00.1: VF 7 failed opcode 6, retval: -5
Guest Error Messages:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES
It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count.
There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue.
The following conditions are required to trigger the bug:
- A port VLAN must be assigned by the host
- The MTU must be set >1500 by the guest
It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes.
[ Impact ]
Intel E810 (host: ice) SR-IOV VFs (guest: iavf) fail when a port VLAN is
assigned by the host and the guest changes the MTU >1500. The guest iavf driver
logs the following error and traffic flow stops:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be used
without this patch.
[ Fix ]
During queue configuration, VF's max packet size was set to
IAVF_MAX_RXBUFFER but on ice max frame size was smaller by VLAN_HLEN
due to making some space for port VLAN as VF is not aware whether it's
in a port VLAN. This mismatch in sizes caused ice to reject queue
configuration with ERR_PARAM error. Proper max_mtu is sent from ice PF
to VF with GET_VF_RESOURCES msg but VF does not look at this.
In iavf change max_frame from IAVF_MAX_RXBUFFER to max_mtu
received from pf with GET_VF_RESOURCES msg to make vf's
max_frame_size dependent from pf. Add check if received max_mtu is
not in eligible range then set it to IAVF_MAX_RXBUFFER.
[ Test Plan ]
1. Create a focal VM and assign an Intel E810 (ice) SR-IOV VF with a port
vlan
Openstack works, as does creating a VM directly with uvtool/libvirt.
The syntax inside <interface> for the port VLAN is <vlan><tag id=‘998’/></vlan>
2. Set an IP on the interface, set it’s link up and verify operation
sudo ip a a 192.168.1.7/24 dev enp7s0
sudo ip link set up dev enp7s0
ping 192.168.1.1
3. Set the MTU to 9000
sudo ip link set mtu 9000 dev enp7s0
4. Test if the interface still works and whether the error was observed:
iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
[ Where problems could occur ]
The iavf driver is unified for multiple host drivers including ice and i40e.
A companion patch for i40e was also submitted as it did not set the max_mtu
although the ice driver already did. We should test the fixed iavf driver
against both i40e (Intel E7xx) and ice (Intel E8xx) cards with and without that
patch.
https://lore.kernel.org/netdev/20220919223428.572091-4-anthony.l.nguyen@intel.com/
[ Other Info ]
Patch was developed by intel, has been accepted into v6.0-rc7 and already
released into upstream stable v5.4.215, v5.15.71 and v5.19.12.
Early SRU is requested as the functionality is required now and there is no
workaround. |
|
| 2022-09-30 16:44:47 |
Trent Lloyd |
description |
Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration.
Environment: Ubuntu OpenStack Focal-Ussuri with OVN
Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE
Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy
Host Error Messages:
ice 0000:98:00.1: VF 7 failed opcode 6, retval: -5
Guest Error Messages:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES
It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count.
There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue.
The following conditions are required to trigger the bug:
- A port VLAN must be assigned by the host
- The MTU must be set >1500 by the guest
It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes.
[ Impact ]
Intel E810 (host: ice) SR-IOV VFs (guest: iavf) fail when a port VLAN is
assigned by the host and the guest changes the MTU >1500. The guest iavf driver
logs the following error and traffic flow stops:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be used
without this patch.
[ Fix ]
During queue configuration, VF's max packet size was set to
IAVF_MAX_RXBUFFER but on ice max frame size was smaller by VLAN_HLEN
due to making some space for port VLAN as VF is not aware whether it's
in a port VLAN. This mismatch in sizes caused ice to reject queue
configuration with ERR_PARAM error. Proper max_mtu is sent from ice PF
to VF with GET_VF_RESOURCES msg but VF does not look at this.
In iavf change max_frame from IAVF_MAX_RXBUFFER to max_mtu
received from pf with GET_VF_RESOURCES msg to make vf's
max_frame_size dependent from pf. Add check if received max_mtu is
not in eligible range then set it to IAVF_MAX_RXBUFFER.
[ Test Plan ]
1. Create a focal VM and assign an Intel E810 (ice) SR-IOV VF with a port
vlan
Openstack works, as does creating a VM directly with uvtool/libvirt.
The syntax inside <interface> for the port VLAN is <vlan><tag id=‘998’/></vlan>
2. Set an IP on the interface, set it’s link up and verify operation
sudo ip a a 192.168.1.7/24 dev enp7s0
sudo ip link set up dev enp7s0
ping 192.168.1.1
3. Set the MTU to 9000
sudo ip link set mtu 9000 dev enp7s0
4. Test if the interface still works and whether the error was observed:
iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
[ Where problems could occur ]
The iavf driver is unified for multiple host drivers including ice and i40e.
A companion patch for i40e was also submitted as it did not set the max_mtu
although the ice driver already did. We should test the fixed iavf driver
against both i40e (Intel E7xx) and ice (Intel E8xx) cards with and without that
patch.
https://lore.kernel.org/netdev/20220919223428.572091-4-anthony.l.nguyen@intel.com/
[ Other Info ]
Patch was developed by intel, has been accepted into v6.0-rc7 and already
released into upstream stable v5.4.215, v5.15.71 and v5.19.12.
Early SRU is requested as the functionality is required now and there is no
workaround. |
Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration.
Environment: Ubuntu OpenStack Focal-Ussuri with OVN
Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE
Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy
Host Error Messages:
ice 0000:98:00.1: VF 7 failed opcode 6, retval: -5
Guest Error Messages:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES
It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count.
There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue.
The following conditions are required to trigger the bug:
- A port VLAN must be assigned by the host
- The MTU must be set >1500 by the guest
It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes.
[ Impact ]
Intel E810 (host: ice) SR-IOV VFs (guest: iavf) fail when a port VLAN is
assigned by the host and the guest changes the MTU >1500. The guest iavf
driver logs the following error and traffic flow stops:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be
used without this patch.
[ Fix ]
During queue configuration, VF's max packet size was set to
IAVF_MAX_RXBUFFER but on ice max frame size was smaller by VLAN_HLEN
due to making some space for port VLAN as VF is not aware whether it's
in a port VLAN. This mismatch in sizes caused ice to reject queue
configuration with ERR_PARAM error. Proper max_mtu is sent from ice PF
to VF with GET_VF_RESOURCES msg but VF does not look at this.
In iavf change max_frame from IAVF_MAX_RXBUFFER to max_mtu
received from pf with GET_VF_RESOURCES msg to make vf's
max_frame_size dependent from pf. Add check if received max_mtu is
not in eligible range then set it to IAVF_MAX_RXBUFFER.
[ Test Plan ]
1. Create a focal VM and assign an Intel E810 (ice) SR-IOV VF with a port
vlan
Openstack works, as does creating a VM directly with uvtool/libvirt.
The syntax inside <interface> for the port VLAN is <vlan><tag id=‘998’/></vlan>
2. Set an IP on the interface, set it’s link up and verify operation
sudo ip a a 192.168.1.7/24 dev enp7s0
sudo ip link set up dev enp7s0
ping 192.168.1.1
3. Set the MTU to 9000
sudo ip link set mtu 9000 dev enp7s0
4. Test if the interface still works and whether the error was observed:
iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
[ Where problems could occur ]
The iavf driver is unified for multiple host drivers including ice and i40e.
A companion patch for i40e was also submitted as it did not set the
max_mtu although the ice driver already did. We should test the fixed iavf
driver against both i40e (Intel E7xx) and ice (Intel E8xx) cards with and
without that patch on the host kernel.
https://lore.kernel.org/netdev/20220919223428.572091-4-anthony.l.nguyen@intel.com/
[ Other Info ]
Patch was developed by intel, has been accepted into v6.0-rc7 and already
released into upstream stable v5.4.215, v5.15.71 and v5.19.12.
Early SRU is requested as the functionality is required now and there is
no workaround. |
|
| 2022-09-30 16:45:04 |
Trent Lloyd |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2022-09-30 16:45:06 |
Trent Lloyd |
linux (Ubuntu): importance |
Undecided |
High |
|
| 2022-10-03 10:15:34 |
Matthew Ruffell |
bug |
|
|
added subscriber Matthew Ruffell |
| 2022-10-04 04:24:29 |
Matthew Ruffell |
summary |
SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) |
iavf: SR-IOV VFs error with no traffic flow when MTU greater than 1500 |
|
| 2022-10-04 04:24:36 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Focal |
|
| 2022-10-04 04:24:36 |
Matthew Ruffell |
bug task added |
|
linux (Ubuntu Focal) |
|
| 2022-10-04 04:24:36 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Kinetic |
|
| 2022-10-04 04:24:36 |
Matthew Ruffell |
bug task added |
|
linux (Ubuntu Kinetic) |
|
| 2022-10-04 04:24:36 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Jammy |
|
| 2022-10-04 04:24:36 |
Matthew Ruffell |
bug task added |
|
linux (Ubuntu Jammy) |
|
| 2022-10-04 04:24:43 |
Matthew Ruffell |
linux (Ubuntu Kinetic): status |
Confirmed |
In Progress |
|
| 2022-10-04 04:24:45 |
Matthew Ruffell |
linux (Ubuntu Jammy): status |
New |
In Progress |
|
| 2022-10-04 04:24:48 |
Matthew Ruffell |
linux (Ubuntu Focal): status |
New |
In Progress |
|
| 2022-10-04 04:24:50 |
Matthew Ruffell |
linux (Ubuntu Focal): importance |
Undecided |
High |
|
| 2022-10-04 04:24:52 |
Matthew Ruffell |
linux (Ubuntu Jammy): importance |
Undecided |
High |
|
| 2022-10-04 04:24:55 |
Matthew Ruffell |
linux (Ubuntu Focal): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2022-10-04 04:24:57 |
Matthew Ruffell |
linux (Ubuntu Jammy): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2022-10-04 04:26:46 |
Matthew Ruffell |
description |
Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration.
Environment: Ubuntu OpenStack Focal-Ussuri with OVN
Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE
Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy
Host Error Messages:
ice 0000:98:00.1: VF 7 failed opcode 6, retval: -5
Guest Error Messages:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES
It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count.
There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue.
The following conditions are required to trigger the bug:
- A port VLAN must be assigned by the host
- The MTU must be set >1500 by the guest
It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes.
[ Impact ]
Intel E810 (host: ice) SR-IOV VFs (guest: iavf) fail when a port VLAN is
assigned by the host and the guest changes the MTU >1500. The guest iavf
driver logs the following error and traffic flow stops:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be
used without this patch.
[ Fix ]
During queue configuration, VF's max packet size was set to
IAVF_MAX_RXBUFFER but on ice max frame size was smaller by VLAN_HLEN
due to making some space for port VLAN as VF is not aware whether it's
in a port VLAN. This mismatch in sizes caused ice to reject queue
configuration with ERR_PARAM error. Proper max_mtu is sent from ice PF
to VF with GET_VF_RESOURCES msg but VF does not look at this.
In iavf change max_frame from IAVF_MAX_RXBUFFER to max_mtu
received from pf with GET_VF_RESOURCES msg to make vf's
max_frame_size dependent from pf. Add check if received max_mtu is
not in eligible range then set it to IAVF_MAX_RXBUFFER.
[ Test Plan ]
1. Create a focal VM and assign an Intel E810 (ice) SR-IOV VF with a port
vlan
Openstack works, as does creating a VM directly with uvtool/libvirt.
The syntax inside <interface> for the port VLAN is <vlan><tag id=‘998’/></vlan>
2. Set an IP on the interface, set it’s link up and verify operation
sudo ip a a 192.168.1.7/24 dev enp7s0
sudo ip link set up dev enp7s0
ping 192.168.1.1
3. Set the MTU to 9000
sudo ip link set mtu 9000 dev enp7s0
4. Test if the interface still works and whether the error was observed:
iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
[ Where problems could occur ]
The iavf driver is unified for multiple host drivers including ice and i40e.
A companion patch for i40e was also submitted as it did not set the
max_mtu although the ice driver already did. We should test the fixed iavf
driver against both i40e (Intel E7xx) and ice (Intel E8xx) cards with and
without that patch on the host kernel.
https://lore.kernel.org/netdev/20220919223428.572091-4-anthony.l.nguyen@intel.com/
[ Other Info ]
Patch was developed by intel, has been accepted into v6.0-rc7 and already
released into upstream stable v5.4.215, v5.15.71 and v5.19.12.
Early SRU is requested as the functionality is required now and there is
no workaround. |
BugLink: https://bugs.launchpad.net/bugs/1983656
[Impact]
Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration.
Environment: Ubuntu OpenStack Focal-Ussuri with OVN
Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE
Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy
Host Error Messages:
ice 0000:98:00.1: VF 7 failed opcode 6, retval: -5
Guest Error Messages:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES
It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count.
There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in 5.15.0-41-generic and did not resolve the issue.
The following conditions are required to trigger the bug:
- A port VLAN must be assigned by the host
- The MTU must be set >1500 by the guest
There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be used without these patches.
[Fix]
iavf currently sets the maximum packet size to IAVF_MAX_RXBUFFER, but on the previous ice driver, it was decremented by VLAN_HLEN to make some space to fit the VLAN header. This doesn't happen on iavf, and we end up trying to use a packet size larger than IAVF_MAX_RXBUFFER, causing the IAVF_ERR_PARAM error.
The fix is to change the maximum packet size from IAVF_MAX_RXBUFFER to max_mtu received from the PF via GET_VF_RESOURCES msg.
Also pick up a necessary commit for i40e to announce the correct maximum packet size by GET_VF_RESOURCES msg.
This has been fixed by the following commits:
commit 399c98c4dc50b7eb7e9f24da7ffdda6f025676ef
Author: Michal Jaron <michalx.jaron@intel.com>
Date: Tue Sep 13 15:38:35 2022 +0200
Subject: iavf: Fix set max MTU size with port VLAN and jumbo frames
Link: https://github.com/torvalds/linux/commit/399c98c4dc50b7eb7e9f24da7ffdda6f025676ef
commit 372539def2824c43b6afe2403045b140f65c5acc
Author: Michal Jaron <michalx.jaron@intel.com>
Date: Tue Sep 13 15:38:36 2022 +0200
Subject: i40e: Fix VF set max MTU size
Link: https://github.com/torvalds/linux/commit/372539def2824c43b6afe2403045b140f65c5acc
A test kernel is available in the following ppa:
https://launchpad.net/~arif-ali/+archive/ubuntu/sf00343742
If you install the test kernel to a compute host and VM, when you attach a VF and set the MTU to 9000, it succeeds, and traffic can flow.
[Test Plan]
1. Create a Focal VM and assign an Intel E810 (ice) SR-IOV VF with a port vlan:
Openstack works, as does creating a VM directly with uvtool/libvirt.
$ uvt-kvm create focal-test release=focal
Using the document to understand SRIOV basics in the link below
https://www.intel.com/content/www/us/en/developer/articles/technical/configure-sr-iov-network-virtual-functions-in-linux-kvm.html
The following command show all the bus info for all the network devices
$ lshw -c network -businfo
Choose one, as shown below
pci@0000:17:01.4 ens2f0v4 network Ethernet Adaptive Virtual Function
We can then add the following into the XML definition via “virsh edit focal-test”
<interface type='hostdev' managed='yes'>
<source>
<address type='pci' domain='0x0000' bus='0x17' slot='0x01' function='0x4'/>
</source>
<vlan>
<tag id='998'/>
</vlan>
</interface>
Then we stop and start the VM via "virsh shutdown focal-test" and then "virsh start focal-test". We can then login to the VM using the command below
$ uvt-kvm ssh focal-test
Once you have logged in, run the following ip parameters
$ sudo ip a a 192.168.1.7/24 dev enp7s0
$ sudo ip link set up dev enp7s0
$ sudo ip link set mtu 9000 dev enp7s0
Now check dmesg, and we will find the error
[ 61.529605] iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
Setting the IP and bringing the link up
[ 36.228877] iavf 0000:07:00.0 enp7s0: NIC Link is Up Speed is 25 Gbps Full Duplex
[ 36.228887] IPv6: ADDRCONF(NETDEV_CHANGE): enp7s0: link becomes ready
[ 45.740100] crng init done
[ 45.740102] random: 7 urandom warning(s) missed due to ratelimiting
Then setting the MTU
[ 61.433706] iavf 0000:07:00.0: Received 16 queues, but can only have a max of 4
[ 61.433707] iavf 0000:07:00.0: Fixing by reducing queues to 4
[ 61.529605] iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
[ 61.552890] iavf 0000:07:00.0 enp7s0: NIC Link is Up Speed is 25 Gbps Full Duplex
There is a test kernel available in the following ppa:
https://launchpad.net/~arif-ali/+archive/ubuntu/sf00343742
If you install the test kernel, setting the MTU to 9000 works as expected and traffic can flow.
[Where problems could occur]
We are changing how maximum MTU is calculated and applied to VFs in the iavf and i40e drivers. Currently, any MTU over 1500 does not work at all when a port VLAN is enabled, but if someone has somehow got their setup to work, they may see a difference in MTU with these patches applied.
The iavf and i40e drivers are a popular driver, and if a regression were to occur, initialisation and bringup of these network devices and VFs might fail.
Most users currently using MTUs of 1500 are unlikely to see any difference or be at risk of regression.
[Other Info]
Both patches were developed by intel, and have been accepted into v6.0-rc7 and are already released into upstream stable v5.4.215, v5.15.71 and v5.19.12. These patches are well tested by the community and considered safe. |
|
| 2022-10-04 04:27:21 |
Matthew Ruffell |
description |
BugLink: https://bugs.launchpad.net/bugs/1983656
[Impact]
Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration.
Environment: Ubuntu OpenStack Focal-Ussuri with OVN
Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE
Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy
Host Error Messages:
ice 0000:98:00.1: VF 7 failed opcode 6, retval: -5
Guest Error Messages:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES
It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count.
There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in 5.15.0-41-generic and did not resolve the issue.
The following conditions are required to trigger the bug:
- A port VLAN must be assigned by the host
- The MTU must be set >1500 by the guest
There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be used without these patches.
[Fix]
iavf currently sets the maximum packet size to IAVF_MAX_RXBUFFER, but on the previous ice driver, it was decremented by VLAN_HLEN to make some space to fit the VLAN header. This doesn't happen on iavf, and we end up trying to use a packet size larger than IAVF_MAX_RXBUFFER, causing the IAVF_ERR_PARAM error.
The fix is to change the maximum packet size from IAVF_MAX_RXBUFFER to max_mtu received from the PF via GET_VF_RESOURCES msg.
Also pick up a necessary commit for i40e to announce the correct maximum packet size by GET_VF_RESOURCES msg.
This has been fixed by the following commits:
commit 399c98c4dc50b7eb7e9f24da7ffdda6f025676ef
Author: Michal Jaron <michalx.jaron@intel.com>
Date: Tue Sep 13 15:38:35 2022 +0200
Subject: iavf: Fix set max MTU size with port VLAN and jumbo frames
Link: https://github.com/torvalds/linux/commit/399c98c4dc50b7eb7e9f24da7ffdda6f025676ef
commit 372539def2824c43b6afe2403045b140f65c5acc
Author: Michal Jaron <michalx.jaron@intel.com>
Date: Tue Sep 13 15:38:36 2022 +0200
Subject: i40e: Fix VF set max MTU size
Link: https://github.com/torvalds/linux/commit/372539def2824c43b6afe2403045b140f65c5acc
A test kernel is available in the following ppa:
https://launchpad.net/~arif-ali/+archive/ubuntu/sf00343742
If you install the test kernel to a compute host and VM, when you attach a VF and set the MTU to 9000, it succeeds, and traffic can flow.
[Test Plan]
1. Create a Focal VM and assign an Intel E810 (ice) SR-IOV VF with a port vlan:
Openstack works, as does creating a VM directly with uvtool/libvirt.
$ uvt-kvm create focal-test release=focal
Using the document to understand SRIOV basics in the link below
https://www.intel.com/content/www/us/en/developer/articles/technical/configure-sr-iov-network-virtual-functions-in-linux-kvm.html
The following command show all the bus info for all the network devices
$ lshw -c network -businfo
Choose one, as shown below
pci@0000:17:01.4 ens2f0v4 network Ethernet Adaptive Virtual Function
We can then add the following into the XML definition via “virsh edit focal-test”
<interface type='hostdev' managed='yes'>
<source>
<address type='pci' domain='0x0000' bus='0x17' slot='0x01' function='0x4'/>
</source>
<vlan>
<tag id='998'/>
</vlan>
</interface>
Then we stop and start the VM via "virsh shutdown focal-test" and then "virsh start focal-test". We can then login to the VM using the command below
$ uvt-kvm ssh focal-test
Once you have logged in, run the following ip parameters
$ sudo ip a a 192.168.1.7/24 dev enp7s0
$ sudo ip link set up dev enp7s0
$ sudo ip link set mtu 9000 dev enp7s0
Now check dmesg, and we will find the error
[ 61.529605] iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
Setting the IP and bringing the link up
[ 36.228877] iavf 0000:07:00.0 enp7s0: NIC Link is Up Speed is 25 Gbps Full Duplex
[ 36.228887] IPv6: ADDRCONF(NETDEV_CHANGE): enp7s0: link becomes ready
[ 45.740100] crng init done
[ 45.740102] random: 7 urandom warning(s) missed due to ratelimiting
Then setting the MTU
[ 61.433706] iavf 0000:07:00.0: Received 16 queues, but can only have a max of 4
[ 61.433707] iavf 0000:07:00.0: Fixing by reducing queues to 4
[ 61.529605] iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
[ 61.552890] iavf 0000:07:00.0 enp7s0: NIC Link is Up Speed is 25 Gbps Full Duplex
There is a test kernel available in the following ppa:
https://launchpad.net/~arif-ali/+archive/ubuntu/sf00343742
If you install the test kernel, setting the MTU to 9000 works as expected and traffic can flow.
[Where problems could occur]
We are changing how maximum MTU is calculated and applied to VFs in the iavf and i40e drivers. Currently, any MTU over 1500 does not work at all when a port VLAN is enabled, but if someone has somehow got their setup to work, they may see a difference in MTU with these patches applied.
The iavf and i40e drivers are a popular driver, and if a regression were to occur, initialisation and bringup of these network devices and VFs might fail.
Most users currently using MTUs of 1500 are unlikely to see any difference or be at risk of regression.
[Other Info]
Both patches were developed by intel, and have been accepted into v6.0-rc7 and are already released into upstream stable v5.4.215, v5.15.71 and v5.19.12. These patches are well tested by the community and considered safe. |
BugLink: https://bugs.launchpad.net/bugs/1983656
[Impact]
Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration.
Environment: Ubuntu OpenStack Focal-Ussuri with OVN
Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE
Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy
Host Error Messages:
ice 0000:98:00.1: VF 7 failed opcode 6, retval: -5
Guest Error Messages:
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES
It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count.
There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in 5.15.0-41-generic and did not resolve the issue.
The following conditions are required to trigger the bug:
- A port VLAN must be assigned by the host
- The MTU must be set >1500 by the guest
There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be used without these patches.
[Fix]
iavf currently sets the maximum packet size to IAVF_MAX_RXBUFFER, but on the previous ice driver, it was decremented by VLAN_HLEN to make some space to fit the VLAN header. This doesn't happen on iavf, and we end up trying to use a packet size larger than IAVF_MAX_RXBUFFER, causing the IAVF_ERR_PARAM error.
The fix is to change the maximum packet size from IAVF_MAX_RXBUFFER to max_mtu received from the PF via GET_VF_RESOURCES msg.
Also pick up a necessary commit for i40e to announce the correct maximum packet size by GET_VF_RESOURCES msg.
This has been fixed by the following commits:
commit 399c98c4dc50b7eb7e9f24da7ffdda6f025676ef
Author: Michal Jaron <michalx.jaron@intel.com>
Date: Tue Sep 13 15:38:35 2022 +0200
Subject: iavf: Fix set max MTU size with port VLAN and jumbo frames
Link: https://github.com/torvalds/linux/commit/399c98c4dc50b7eb7e9f24da7ffdda6f025676ef
commit 372539def2824c43b6afe2403045b140f65c5acc
Author: Michal Jaron <michalx.jaron@intel.com>
Date: Tue Sep 13 15:38:36 2022 +0200
Subject: i40e: Fix VF set max MTU size
Link: https://github.com/torvalds/linux/commit/372539def2824c43b6afe2403045b140f65c5acc
A test kernel is available in the following ppa:
https://launchpad.net/~arif-ali/+archive/ubuntu/sf00343742
If you install the test kernel to a compute host and VM, when you attach a VF and set the MTU to 9000, it succeeds, and traffic can flow.
[Test Plan]
Create a Focal VM and assign an Intel E810 (ice) SR-IOV VF with a port vlan:
Openstack works, as does creating a VM directly with uvtool/libvirt.
$ uvt-kvm create focal-test release=focal
Using the document to understand SRIOV basics in the link below
https://www.intel.com/content/www/us/en/developer/articles/technical/configure-sr-iov-network-virtual-functions-in-linux-kvm.html
The following command show all the bus info for all the network devices
$ lshw -c network -businfo
Choose one, as shown below
pci@0000:17:01.4 ens2f0v4 network Ethernet Adaptive Virtual Function
We can then add the following into the XML definition via “virsh edit focal-test”
<interface type='hostdev' managed='yes'>
<source>
<address type='pci' domain='0x0000' bus='0x17' slot='0x01' function='0x4'/>
</source>
<vlan>
<tag id='998'/>
</vlan>
</interface>
Then we stop and start the VM via "virsh shutdown focal-test" and then "virsh start focal-test". We can then login to the VM using the command below
$ uvt-kvm ssh focal-test
Once you have logged in, run the following ip parameters
$ sudo ip a a 192.168.1.7/24 dev enp7s0
$ sudo ip link set up dev enp7s0
$ sudo ip link set mtu 9000 dev enp7s0
Now check dmesg, and we will find the error
[ 61.529605] iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
Setting the IP and bringing the link up
[ 36.228877] iavf 0000:07:00.0 enp7s0: NIC Link is Up Speed is 25 Gbps Full Duplex
[ 36.228887] IPv6: ADDRCONF(NETDEV_CHANGE): enp7s0: link becomes ready
[ 45.740100] crng init done
[ 45.740102] random: 7 urandom warning(s) missed due to ratelimiting
Then setting the MTU
[ 61.433706] iavf 0000:07:00.0: Received 16 queues, but can only have a max of 4
[ 61.433707] iavf 0000:07:00.0: Fixing by reducing queues to 4
[ 61.529605] iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
[ 61.552890] iavf 0000:07:00.0 enp7s0: NIC Link is Up Speed is 25 Gbps Full Duplex
There is a test kernel available in the following ppa:
https://launchpad.net/~arif-ali/+archive/ubuntu/sf00343742
If you install the test kernel, setting the MTU to 9000 works as expected and traffic can flow.
[Where problems could occur]
We are changing how maximum MTU is calculated and applied to VFs in the iavf and i40e drivers. Currently, any MTU over 1500 does not work at all when a port VLAN is enabled, but if someone has somehow got their setup to work, they may see a difference in MTU with these patches applied.
The iavf and i40e drivers are a popular driver, and if a regression were to occur, initialisation and bringup of these network devices and VFs might fail.
Most users currently using MTUs of 1500 are unlikely to see any difference or be at risk of regression.
[Other Info]
Both patches were developed by intel, and have been accepted into v6.0-rc7 and are already released into upstream stable v5.4.215, v5.15.71 and v5.19.12. These patches are well tested by the community and considered safe. |
|
| 2022-10-04 04:50:34 |
Matthew Ruffell |
tags |
servcert-472 sts |
focal jammy kinetic servcert-472 sts |
|
| 2022-10-05 13:19:23 |
Roman Lapszow |
bug |
|
|
added subscriber Roman Lapszow |
| 2022-10-12 09:10:14 |
Stefan Bader |
linux (Ubuntu Kinetic): status |
In Progress |
Fix Committed |
|
| 2022-10-12 09:10:40 |
Stefan Bader |
linux (Ubuntu Jammy): status |
In Progress |
Fix Committed |
|
| 2022-10-12 09:11:09 |
Stefan Bader |
linux (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2022-10-25 09:48:16 |
Arif Ali |
tags |
focal jammy kinetic servcert-472 sts |
focal jammy kinetic servcert-472 sts verification-done-focal |
|
| 2022-10-26 15:17:43 |
Arif Ali |
tags |
focal jammy kinetic servcert-472 sts verification-done-focal |
focal jammy kinetic servcert-472 sts verification-done-focal verification-done-jammy |
|
| 2022-11-15 10:47:50 |
Launchpad Janitor |
linux (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2022-11-15 10:47:50 |
Launchpad Janitor |
cve linked |
|
2022-2978 |
|
| 2022-11-15 10:47:50 |
Launchpad Janitor |
cve linked |
|
2022-29901 |
|
| 2022-11-15 10:47:50 |
Launchpad Janitor |
cve linked |
|
2022-3028 |
|
| 2022-11-15 10:47:50 |
Launchpad Janitor |
cve linked |
|
2022-40768 |
|
| 2022-11-15 10:47:50 |
Launchpad Janitor |
cve linked |
|
2022-42719 |
|
| 2022-11-15 10:52:08 |
Launchpad Janitor |
linux (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2022-12-07 11:38:23 |
Ubuntu Kernel Bot |
tags |
focal jammy kinetic servcert-472 sts verification-done-focal verification-done-jammy |
focal jammy kernel-spammed-kinetic-linux kinetic servcert-472 sts verification-done-focal verification-done-jammy verification-needed-kinetic |
|
| 2022-12-07 15:49:50 |
Ubuntu Kernel Bot |
tags |
focal jammy kernel-spammed-kinetic-linux kinetic servcert-472 sts verification-done-focal verification-done-jammy verification-needed-kinetic |
focal jammy kernel-spammed-jammy-linux-bluefield kernel-spammed-kinetic-linux kinetic servcert-472 sts verification-done-focal verification-needed-jammy verification-needed-kinetic |
|
| 2022-12-12 12:47:02 |
Ubuntu Kernel Bot |
tags |
focal jammy kernel-spammed-jammy-linux-bluefield kernel-spammed-kinetic-linux kinetic servcert-472 sts verification-done-focal verification-needed-jammy verification-needed-kinetic |
focal jammy kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kinetic servcert-472 sts verification-done-focal verification-needed-jammy verification-needed-kinetic |
|
| 2022-12-14 03:06:49 |
Ubuntu Kernel Bot |
tags |
focal jammy kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kinetic servcert-472 sts verification-done-focal verification-needed-jammy verification-needed-kinetic |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
|
| 2022-12-15 13:52:17 |
Ubuntu Kernel Bot |
tags |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
|
| 2023-01-06 00:35:21 |
Launchpad Janitor |
linux (Ubuntu Kinetic): status |
Fix Committed |
Fix Released |
|
| 2023-01-09 22:27:00 |
Ubuntu Kernel Bot |
tags |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kernel-spammed-kinetic-linux-ibm kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
|
| 2023-01-12 16:42:28 |
Ubuntu Kernel Bot |
tags |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kernel-spammed-kinetic-linux-ibm kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kernel-spammed-kinetic-linux-aws kernel-spammed-kinetic-linux-ibm kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
|
| 2023-01-17 13:33:48 |
Ubuntu Kernel Bot |
tags |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kernel-spammed-kinetic-linux-aws kernel-spammed-kinetic-linux-ibm kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kernel-spammed-kinetic-linux-aws kernel-spammed-kinetic-linux-azure kernel-spammed-kinetic-linux-ibm kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
|
| 2023-01-31 23:26:55 |
Ubuntu Kernel Bot |
tags |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kernel-spammed-kinetic-linux-aws kernel-spammed-kinetic-linux-azure kernel-spammed-kinetic-linux-ibm kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-aws-5.19 kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kernel-spammed-kinetic-linux-aws kernel-spammed-kinetic-linux-azure kernel-spammed-kinetic-linux-ibm kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
|
| 2023-01-31 23:28:04 |
Ubuntu Kernel Bot |
tags |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-aws-5.19 kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kernel-spammed-kinetic-linux-aws kernel-spammed-kinetic-linux-azure kernel-spammed-kinetic-linux-ibm kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
focal jammy kernel-spammed-focal-linux-azure kernel-spammed-focal-linux-xilinx-zynqmp kernel-spammed-jammy-linux-aws-5.19 kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-lowlatency-hwe-5.19 kernel-spammed-jammy-linux-nvidia kernel-spammed-kinetic-linux kernel-spammed-kinetic-linux-aws kernel-spammed-kinetic-linux-azure kernel-spammed-kinetic-linux-ibm kinetic servcert-472 sts verification-needed-focal verification-needed-jammy verification-needed-kinetic |
|
