Mellanox mlx4_en network driver is not automatically loaded

Similar/same issue in debian-installer if that helps: https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1015339

It seems there are 2 issues here:
a.) we have a general issue with the initramfs shipped with our ephemeral images. We're intending that they boot and load iscsi over network, but we do not build the initramfs with 'MODULES=netboot' (initramfs.conf(5)).
b.) it seems there is a specific issue with the mellanox ethernet drivers at least as shipped in 12.04 in that they do not autoload correctly. The work around there is to re-build the initramfs with the specific drivers listed in /etc/initramfs-tools/modules .

Re: #2: The issue is that the ml4x_en module is not being autoloaded. I suspsect that this is a symbol's issue, but I don't have enough context.

After a machine has been installed, by correcting the module autoload in the initrd, the following udev rules are created:

$ cat /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

[...]

# PCI device 0x15b3:/sys/devices/pci0000:00/0000:00:05.0/0000:05:00.0 (mlx4_core)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="78:e7:d1:23:2a:55", ATTR{dev_id}=="0x1", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

This could be further evidence that the core module is incorrectly being loaded for the ethernet device?

agy, that seems like a good diagnosis.

I've went ahead and added a kernel task for this.
So the work here is:
 * ml4x_en kernel module needs to specify pci-ids. currently the ml4x_core specifies pci-ids, so it is what gets automatically loaded.
 * kernel needs to add ml4x_net to the list so it gets collected as a network driver (bug 1015339)
 * maas ephemeral image build process needs to build initramfs with 'MODULES=netboot'
 * maas ephemeral image build process will create initramfs that works around this specific network driver issue.

so it seems less straightforward than i had originally thought. the kernel modules don't automatically load. that seems by design "Mellanox OFED Stack for Linux User’s Manual" [1]. So i'm not sure how the installer (or maas) should know that this should happen. The mlx4_core can be used both by infiniband driver and mlx4_en.

[1] http://ibicluster.uchicago.edu/roll-documentation/mlnxUserGuides/Mellanox_OFED_Linux_user_manual.pdf

Hi Guys,

According to the user manual described by smoser:

By default, the mlx4_en (Ethernet) driver is not loaded. To change this, edit the file /etc/
infiniband/openib.conf and modify the MLX4_EN_LOAD parameter as follows:
# Load MLX4_EN module
MLX4_EN_LOAD=yes

This is for RH though...

Leo

Discussion in IRC at [1], with the following pulled from the
"Mellanox OFED Stack for Linux User's Manual" [2]:
 | mlx4_core
 | Handles low-level functions like device initialization and firmware
 | commands processing. Also controls resource allocation so th at the
 | InfiniBand and Ethernet functions can share the device without
 | interfering with each other.

So, it would appear that the modules should load correctly together, and
the easiest solution is then to just load both, possibly through a udev
rule. Basically, just if mlx4_core is loaded, load the _en and _ib.
--
[1] http://irclogs.ubuntu.com/2013/02/05/%23ubuntu-devel.html#t17:39
[2] http://ibicluster.uchicago.edu/roll-documentation/mlnxUserGuides/Mellanox_OFED_Linux_user_manual.pdf

more discussion in IRC, slangasek suggests he'd prefer modprobe.d configuration.
The attached branches do that.

OK. so, I basically had 2 choices for modprobe.d auto_loading of mlx4_en upon usage of mlx4_core. Neither was perfect.

A.) "softdep mlx4_core post: mlx4_en mlx4_ib"
The softdep route hits what it seems to me to be a bug in 12.04 in modprobe
or update-initramfs. The updating of initramfs gives warnings like:
  FATAL: modprobe: softdep dependency loop encountered inserting mlx4_core
  FATAL: modprobe: softdep dependency loop encountered inserting mlx4_core
It exits with success (0), but it does not put mlx4_en inside the initramfs.

B.) "install mlx4_core /sbin/modprobe --ignore-install mlx4_core; /sbin/modprobe mlx4_en; /sbin/modprobe mlx4_ib"

With this entry, creation of initramfs goes fine, but but if there is no mlx4_en or mlx4_ib module available upon request to load the mlx4_core module, the user gets a confusing error:
   $ sudo modprobe mlx4_core; echo $?
   FATAL: Could not read '/lib/modules/3.2.0-37-virtual/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko': No such file or directory
   0

It really feels to me that soft-dep is more intended to satisfy this exact need, but the initramfs is a killer. So, we'll go with the 'install' route.

### issue with softdep route ####

$ sudo rm /boot/initrd.img-$(uname -r); sudo update-initramfs -u -k $(uname -r); echo $?; echo === ; cat /etc/modprobe.d/mlx4.conf; echo ===; lsinitramfs /boot/initrd.img-$(uname -r) | grep mlx
update-initramfs: Generating /boot/initrd.img-3.2.0-37-virtual
FATAL: modprobe: softdep dependency loop encountered inserting mlx4_core
FATAL: modprobe: softdep dependency loop encountered inserting mlx4_core
0
===
# mlx4_core should load mlx4_en (LP: #1115710).
softdep mlx4_core post: mlx4_en mlx4_ib
===
lib/modules/3.2.0-37-virtual/kernel/drivers/net/ethernet/mellanox/mlx4
lib/modules/3.2.0-37-virtual/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko
etc/modprobe.d/mlx4.conf

### issue with install route ####

$ sudo cp ~/mlx4.conf.install /etc/modprobe.d/mlx4.conf; sudo rmmod mlx4_en mlx4_ib mlx4_core &>/dev/null; sudo rm -f /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko; echo === ; cat /etc/modprobe.d/mlx4.conf; echo ===; sudo modprobe mlx4_core; echo $?;
===
# mlx4_core should load mlx4_en (LP: #1115710).
#softdep mlx4_core post: mlx4_en mlx4_ib
install mlx4_core /sbin/modprobe --ignore-install mlx4_core; /sbin/modprobe mlx4_en; /sbin/modprobe mlx4_ib
===
FATAL: Could not read '/lib/modules/3.2.0-37-virtual/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko': No such file or directory
0

2 other bits of info here
a.) Thanks to entropyworks, someone has actually verified this with hardware present
b.) after adding or removing /etc/modprobe.d files (or /etc/modules) you really need to update-initramfs. This is because if the module gets loaded in the initramfs without the 'install' rule that we're putting in place, the new rule will not have affect.

Conversation today with infinity in #ubuntu-devel http://irclogs.ubuntu.com/2013/02/07/%23ubuntu-devel.html .
He suggested removing the auto-loading of the mlx4_ib at least for now. That avoids pulling infiniband stack into initramfs.

Also, he pointed out that kmod (raring) does not have the issue with update-initramfs.
So we'll use the soft-dep in raring and the modprobe in quantal/precise.

raring fix will be in kmod, not in module-init-tools (so invalid for raring/module-init-tools)

I'm marking maas as 'fix-committed'. the maas ephemeral image builder now:
 * works around the lack of mlx4.conf if it is not present (so mlx4_en will be loaded in initramfs)
 * builds initramfs with MODULES=netboot.

daily ephemeral images should start to pick those changes up.

There are now ephemeral images at http://maas.ubuntu.com/images/ephemeral/daily/raring/20130207/
that should have the mellanox driver inside the initramfs.

This bug was fixed in the package kmod - 9-2ubuntu7

---------------
kmod (9-2ubuntu7) raring; urgency=low

  * Add /etc/modprobe.d/mlx4.conf for loading mlx4_en when mlx4_core
    gets loaded (LP: #1115710).
 -- Scott Moser <email address hidden> Tue, 05 Feb 2013 15:36:49 -0500

I'd call this "fix-committed" until we have a 'released' ephemeral image that contains the changes. You're welcome to dis-agree, but I consider daily builds kind of like "trunk commits".

Not sure this deserves a bug on maas at all, actually, since there's nothing in the maas codebase to change. However, can it be made fix released now?

quantal has seen the end of its life and is no longer receiving any updates. Marking the quantal task for this ticket as "Won't Fix".

Hi,

This Fix is for old releases that doesn't include mlx4_en in their initramfs.
Now, that the initrd include the mlx4_en, so no need to creating a mlx4.conf file with softdep mlx4_core post: mlx4_en
After removing this file we see that mlx4_en loaded after boot due to mlx4_core request a module according to the port type -

drivers/net/ethernet/mellanox/mlx4/main.c
static void mlx4_request_modules(struct mlx4_dev *dev)
{
        int port;
        int has_ib_port = false;
        int has_eth_port = false;
#define EN_DRV_NAME "mlx4_en"
#define IB_DRV_NAME "mlx4_ib"

        for (port = 1; port <= dev->caps.num_ports; port++) {
                if (dev->caps.port_type[port] == MLX4_PORT_TYPE_IB)
                        has_ib_port = true;
                else if (dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH)
                        has_eth_port = true;
        }

        if (has_eth_port)
                request_module_nowait(EN_DRV_NAME);
        if (has_ib_port || (dev->caps.flags & MLX4_DEV_CAP_FLAG_IBOE))
                request_module_nowait(IB_DRV_NAME);
}

We are debianize rdma-utils package that create a file /etc/modpobe.d/mlx4.conf, according to Debian policy package shouldn't modify files installed by another package in any way, it's forbidden.
in addition the rdma service create the same file name and add the following
install mlx4_core /sbin/modprobe --ignore-install mlx4_core && (if [ -f /usr/libexec/mlx4-setup.sh -a -f /etc/rdma/mlx4.conf ]; then /usr/libexec/mlx4-setup.sh < /etc/rdma/mlx4.conf; fi; /sbin/modprobe mlx4_en; /sbin/modprobe mlx4_ib)
this should load the diver with the requested port type configuration.

could you please revert this fix?

Thanks,
Talat

Ping, this migt be causing issue when using the mlx4 driver today.

Ubuntu 16.04 Xenial
4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

The following file is no longer needed, mlx4_core and mlx4_en will be loaded properly automatic
cat /etc/modprobe.d/mlx4.conf
# mlx4_core should load mlx4_en (LP: #1115710).
softdep mlx4_core post: mlx4_en

Another issue I have, I cannot enable VXLAN offloading on boot.
I have tried:

* Keeping /etc/modprobe.d/mlx4.conf and adding "options mlx4_core log_num_mgm_entry_size=-1" to start of file
* Keeping /etc/modprobe.d/mlx4.conf and adding "options mlx4_core log_num_mgm_entry_size=-1" to end of file
* Remove all lines in /etc/modprobe.d/mlx4.conf except for "options mlx4_core log_num_mgm_entry_size=-1"
* Keeping /etc/modprobe.d/mlx4.conf and adding "options mlx4_core log_num_mgm_entry_size=-1" to /etc/modprobe.d/mlx4_core.conf
* Removing /etc/modprobe.d/mlx4.conf and adding "options mlx4_core log_num_mgm_entry_size=-1" to /etc/modprobe.d/mlx4_core.conf

None of these makes the VXLAN offloading capability available.
ethtool -k enp59s0 | grep tx-udp
tx-udp_tnl-segmentation: off [fixed]

But if I remove and load the driver manually after boot and add it again to works.
modprobe -r mlx4_en
modprobe -r mlx4_core

(after I load modules manually below I also get the mlx4_ib module, so the process to remove again involves unloading mlx4_en and mlx4_ib before unloading mlx4_core)

* Keeping /etc/modprobe.d/mlx4.conf and adding "options mlx4_core log_num_mgm_entry_size=-1" to start of file
modprobe mlx4_en
ethtool -k enp59s0 | grep tx-udp
tx-udp_tnl-segmentation: on

* Keeping /etc/modprobe.d/mlx4.conf and adding "options mlx4_core log_num_mgm_entry_size=-1" to end of file
modprobe mlx4_en
ethtool -k enp59s0 | grep tx-udp
tx-udp_tnl-segmentation: on

* Remove all lines in /etc/modprobe.d/mlx4.conf except for "options mlx4_core log_num_mgm_entry_size=-1"
modprobe mlx4_en
ethtool -k enp59s0 | grep tx-udp
tx-udp_tnl-segmentation: on

* Removing /etc/modprobe.d/mlx4.conf and adding "options mlx4_core log_num_mgm_entry_size=-1" to /etc/modprobe.d/mlx4_core.conf
modprobe mlx4_en
ethtool -k enp59s0 | grep tx-udp
tx-udp_tnl-segmentation: on

On boot the following modules is loaded:
# lsmod | grep mlx
mlx4_en 110592 0
vxlan 49152 2 i40e,mlx4_en
mlx4_core 286720 1 mlx4_en
ptp 20480 3 igb,i40e,mlx4_en

When I modprobe manually the following modules is loaded:
# lsmod | grep mlx
mlx4_ib 151552 0
mlx4_en 110592 0
mlx4_core 286720 2 mlx4_en,mlx4_ib
ib_sa 36864 3 rdma_cm,ib_cm,mlx4_ib
ib_mad 49152 3 ib_cm,ib_sa,mlx4_ib
ib_core 106496 7 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,ib_mad,ib_iser
vxlan 49152 2 i40e,mlx4_en
ptp 20480 3 igb,i40e,mlx4_en

What could cause this blockage that the driver is not loaded properly with the options on boot?
Best regards

Must have had a brain freeze or something, I rebuild my initramfs with belo and it worked after boot.
update-initramfs -k `uname -r` -c

The point that the default /etc/modprobe.d/mlx4.conf should be removed still stands, I'll report a new bug to kmod I guess.

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release