Intel x550 not negotiating to 10Gb on x299 chipset

Bug #1812410 reported by Kevin
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

I am having an issue with this particular machine auto negotiating 10Gb. The onboard LOM will only negotiate to 1Gb. If I run `ethtool -s <device> advertise 0x1000`, the machine will delink and then negotiate to 10Gb and work as expecting. Rebooting the machine (or unplugging network cable) will cause it to negotiate to 1G again resulting in having to run the ethtool command again.

This is what I have tried:
Tried different cable from confirmed working machine that auto negotiates to 10Gb.
Tried upgrading ixgbe driver to latest (5.5.3)
Tried upgrading X550 firmware.
I put an Intel x540-T1 card in the machine and that card works properly and negotiates to 10Gb automatically. Move the cable to the LOM x550, and it still negotiates to 1Gb.
I also tried loading a live version of Fedora 29 (Kernel 4.18) with same exact symptoms.

We have several other machine negotiating fine on the same exact switch. I have tried those cables to test this problematic machine with same result as above.

We have a machine that is pretty similar in hardware that is not exhibiting this issue. It is an ASUS x99 motherboard, with the same LOM Intel x550 card, running on 16.04.

The machine that is causing problems is an Asus x299 motherboard with the Intel x550 LOM card running 18.04 (16.04 won't work on the x299 chipset for some reason). I have a suspicion that it could potentially be related to the x299?

The only way I have been able to get this problematic machine to negotiate to 10Gbps automatically is by plugging a cable directly from the LOM x550 of the problematic machine to another machines 10Gb network card. Doing that, the problematic machine will negotiate properly to 10Gb.

I'm trying to find another 10Gb switch to test with to see if it could be unique to this combination of switch and machine. However, I'm reluctant to blame it on the switch since we have about several other machine working perfectly fine on the switch (some with the same exact x550 LOM).

Other notes:

https://serverfault.com/questions/949006/intel-x550-not-negotiating-to-10gb?noredirect=1#comment1233436_949006
---
ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-12-14 (34 days ago)
InstallationMedia: Ubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
IwConfig:
 enp179s0f1 no wireless extensions.

 lo no wireless extensions.

 enp179s0f0 no wireless extensions.
MachineType: System manufacturer System Product Name
NonfreeKernelModules: nvidia_modeset nvidia
Package: linux (not installed)
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-43-generic root=UUID=7e94e4fd-ff67-46c2-9b58-3f7bb3840be4 ro quiet splash vt.handoff=1
ProcVersionSignature: Ubuntu 4.15.0-43.46-generic 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-43-generic N/A
 linux-backports-modules-4.15.0-43-generic N/A
 linux-firmware 1.173.3
RfKill:

Tags: bionic
Uname: Linux 4.15.0-43-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 11/30/2018
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 0905
dmi.board.asset.tag: Default string
dmi.board.name: WS X299 SAGE/10G
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr0905:bd11/30/2018:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnWSX299SAGE/10G:rvrRev1.xx:cvnDefaultstring:ct3:cvrDefaultstring:
dmi.product.family: To be filled by O.E.M.
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer

Revision history for this message
Kevin (kvasko) wrote :
Revision history for this message
Kevin (kvasko) wrote :

adding additional log information.

Revision history for this message
Kevin (kvasko) wrote :

adding additional log information

Revision history for this message
Kevin (kvasko) wrote :

adding additional log information

description: updated
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1812410

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Kevin (kvasko) wrote : AlsaInfo.txt

apport information

tags: added: apport-collected bionic
description: updated
Revision history for this message
Kevin (kvasko) wrote : AudioDevicesInUse.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : CRDA.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : Lspci.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : Lsusb.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : ProcModules.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : PulseList.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : UdevDb.txt

apport information

Revision history for this message
Kevin (kvasko) wrote : WifiSyslog.txt

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Kai-Heng Feng (kaihengfeng) wrote :

Would it be possible for you to test the latest upstream kernel? Refer
to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest
v5.0-rc3 kernel[0].

If this bug is fixed in the mainline kernel, please add the following
tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag:
'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as
"Confirmed".

Thanks in advance.

[0] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0-rc3/

Revision history for this message
Kevin (thenewmanium) wrote :

Hi, this issue affects my hosts as well. I have about 40 hosts running Intel X550s that were working fine on kernel 4.4.0-141-generic with regard to negotiating 10Gb automatically. Suddenly after booting into 4.15.0-43-generic they automatically negotiate at 1Gb. I have to use ethtool to force them to 10Gb.

These are all Dell R640 so I don't think the issue is tied to the Asus x299. Seems more tightly correlated with the Intel X550 and the kernel version (4.15+ for me).

I tried the 5.0-rc3 kernel you linked and the issue still exists there. So, somewhere between 4.4 and 4.13 the issue started and it still has not been fixed.

Revision history for this message
Kevin (kvasko) wrote :

@Kevin (thenewmanium).

I honestly think this is a Intel driver bug. See my thread here on the Intel forums.

https://forums.intel.com/s/question/0D50P00004CK4wZSAT/intel-x550-not-negotiating-to-10gb-on-x299-chipset?language=en_US

Their conclusion was that it was an Asus changed x550 chip. Asus said they won't fix it because they "don't support linux". Intel told me they won't do anything because Asus borked with the firmware. I'm basically stuck in the middle of two vendors not wanting to do anything because one doesn't support it and the other doesn't want to look at it because the other vendor changed it.

If you can validate that its the same issue with different hardware, it might get some traction (fingers crossed).

Revision history for this message
Kevin (thenewmanium) wrote :

I'm pushing on Dell to push on Intel. Not getting a lot of traction on that but I'll update if I do. Definitely have this issue on all my Dell R640s with X550s though.

I also suspect the ixgbe module here. The kernel 4.4.0-141 version is 4.2.1. Going to Ubuntu's 4.15.0-43 kernel takes it to 5.1.0 where the 1Gb negotiation issue pops up. I also tried upgrading ixgbe to 5.5.5 and saw the same issue, so it isn't fixed yet or anything.

Revision history for this message
Kevin (kvasko) wrote :

@Kevin (thenewmanium)

Can you run and post the response on your system?

ethtool -i <device>

and

ethtool -k <device>

I'm opening up that ticket I had open with Intel again to see if we can some traction from that.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.