Needed patches for InfiniBand Support: Flow Steering and Offload Support + Fixes

This affects Trusty, Utopic and Vivid (all patches for Trusty and checksum offload support for others). Providing a PPA with patches for Mellanox to test and provide feedback so we can create a SRU proposal.

Thank you

Tinoco

Hello Rafael,

Sorry for the late reply, We tested the PPA and it's good, Could you please create a SRU with it.

Thanks,
Kamal

Thank you Kamal,

I'll be providing the debdiffs and asking for sponsorship/upload for the SRU real soon.

Cheers,

Rafael

I'm providing SRU for *Trusty*:

- libibverbs (also provides: ibverbs-utils) (new version: 1.1.7-1ubuntu2)
- libmlx4 (depends: libibverbs >= 1.1.7-1ubuntu2)
- tgt (from LP: #1445038) (depends: libibverbs1 >= 1.1.7-1ubuntu2, librdmacm1 >= 1.0.16-1ubuntu1)
- librdmawrap2 (qpid-cpp) (from LP: #1367255) (new version: 0.16-7ubuntu6) (depends: libibverbs1 >= 1.1.7-1ubuntu2, librdmacm1 >= 1.0.16-1ubuntu1)

And also providing SRU for Trusty for rdepends:

- librdmacm1 (also provides: rdmacm-utils) (new version: 1.0.16-1ubuntu1) (depends: libibverbs1 >= 1.1.7-1ubuntu2)
- libnes1 (new version: 1.1.3-1ubuntu1) (depends: libibverbs1 >= 1.1.7-1ubuntu2)
- libmthca1 (new version: 1.0.6-1ubuntu2) (depends: libibverbs1 >= 1.1.7-1ubuntu2)
- libmlx5-1 (new version: 1.0.1-0ubuntu3) (depends: libibverbs1 >= 1.1.7-1ubuntu2)
- libipathverbs1 (new version: 1.2-1ubuntu1) (depends: libibverbs1 >= 1.1.7-1ubuntu2)
- libibcm1 (new version: 1.0.4-1.2ubuntu1) (depends: libibverbs1 >= 1.1.7-1ubuntu2)
- libcxgb3-1 (new version: 1.3.1-1ubuntu2) (depends: libibverbs1 >= 1.1.7-1ubuntu2)
- srptools (new version: 0.0.4-1.2ubuntu1) (depends: libibverbs1 >= 1.1.7-1ubuntu2)
- libopenmpi1.6 (openmpi) (new version: 1.6.5-8ubuntu1) (depends: libibverbs1 >= 1.1.7-1ubuntu2)
- perftest (new version: 1.2-OFED-1.4.2-2ubuntu1) (depends: libibverbs1 >= 1.1.7-1ubuntu2, librdmacm1 >= 1.0.16-1ubuntu1)
- fio (new version: 2.1.3-1ubuntu1) (depends: libibverbs1 >= 1.1.7-1ubuntu2, librdmacm1 >= 1.0.16-1ubuntu1)
- glusterfs-common (from glusterfs) (new version: 3.4.2-1ubuntu2) (depends: libibverbs1 >= 1.1.7-1ubuntu2)
- libtotem-pg5 (corosync) (new version: 2.3.3-1ubuntu2) (depends: libibverbs1 >= 1.1.7-1ubuntu2, librdmacm1 >= 1.0.16-1ubuntu1)

Attaching all debdiffs NOW

The attachment "trusty_libibverbs_1.1.7-1ubuntu2.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

I have created the following PPA:

https://launchpad.net/~inaddy/+archive/ubuntu/lp1409904/

(Wait for packages to finish being built).

Containing most packages from this SRU so people could provide feedback AND/OR use a hotfixed environment.

Thank you

Rafael Tinoco

utopic_libibverbs_1.1.8-1ubuntu2.debdiff

utopic_libmlx4_1.0.5-1ubuntu2.debdiff

utopic_tgt_1.0.43-0ubuntu5.debdiff

Can you make the following fixes for the vivid patch series:
1) ensure that every changelog is closing #1409904
2) libibverbs needs to be 1.1.8-1ubuntu1.1
3) tgt needs to be 1.0.43-0ubuntu4.1
4) libmlx4 should be 1.0.6-1ubuntu0.1

Thanks,

Thanks Chris,

Doing that right now and re-attaching debdiffs for Vivid.

Thanks for reviewing all this.

-Rafael

vivid_libibverbs_1.1.8-1ubuntu1.1.debdiff

vivid_libmlx4_1.0.6-1ubuntu0.1.debdiff

vivid_tgt_1.0.43-0ubuntu4.1.debdiff

Per request, adding also Wily debdiffs...

Note that TGT has iSER support since v1.0.44:

inaddy@tmellanox:~/sources-tgt/upstream/tgt/tgt$ git tag --contains 4a4268bc3a458be0190709bdd7f13392f91ca15f
v1.0.44
...
v1.0.57

And

 tgt | 1:1.0.17-1ubuntu2 | precise | source, amd64, armel, armhf, i386, powerpc
 tgt | 1:1.0.43-0ubuntu4 | trusty | source, amd64, arm64, armhf, i386, powerpc, ppc64el
 tgt | 1:1.0.43-0ubuntu4 | utopic | source, amd64, arm64, armhf, i386, powerpc, ppc64el
 tgt | 1:1.0.43-0ubuntu4 | vivid | source, amd64, arm64, armhf, i386, powerpc, ppc64el
 tgt | 1:1.0.43-0ubuntu4 | wily | source, amd64, arm64, armhf, i386, powerpc, ppc64el
 tgt | 1:1.0.57-1ubuntu1 | wily-proposed | source

Wily-proposed already has the 1.0.57-1ubuntu1 version (which contains this fix).

Attaching only libibverbs and libmlx4 debdiffs (checksum offload support capability) per Mellanox request.

wily_libibverbs_1.1.8-1ubuntu2.debdiff

wily_libmlx4_1.0.6-1ubuntu1.debdiff

Uploaded libibverbs/libmlx4 for wily.

I see libibverbs in wily, but not libmlx4. I've ping Chris about that.

It was probably rejected because it has the same version number as the one in the vivid-proposed queue.

This bug was fixed in the package libmlx4 - 1.0.6-1ubuntu2

---------------
libmlx4 (1.0.6-1ubuntu2) wily; urgency=medium

  * Add checksum offload support capability (LP: #1409904)

 -- Rafael David Tinoco (Inaddy) <email address hidden> Mon, 11 May 2015 14:27:50 -0500

This bug was fixed in the package libibverbs - 1.1.8-1ubuntu2

---------------
libibverbs (1.1.8-1ubuntu2) wily; urgency=medium

  * Add checksum offload support capability (LP: #1409904)

 -- Rafael David Tinoco (Inaddy) <email address hidden> Mon, 11 May 2015 14:11:36 -0500

Removed Utopic series since it EOLs in 2 days.

Hello Brian, or anyone else affected,

Accepted libibverbs into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libibverbs/1.1.8-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Brian, or anyone else affected,

Accepted libmlx4 into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libmlx4/1.0.6-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Brian, or anyone else affected,

Accepted tgt into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tgt/1:1.0.43-0ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Status changed to 'Confirmed' because the bug affects multiple users.

After talking to Chris Arges (arges) and Adam Conrad (infinity):

It was clear that if libibverbs fixes (HW enablement) break ABI
compatibility then they are not suitable for a Trusty SRU.

I'll break those fixes and return all of them to their respective cases
since there won't be any ABI being broken:

--

1) TGT daemon supporting iSER (iSCSI over IB) fixes (WILL BE FIXED HERE
since Vivid & Wily were already fixed here):

- LP: #1401575 - iSER not working in Trusty - Invalid.
- LP: #1445038 - TGT does not support ISER discovery - Duplicate of this bug.
- LP: #1443648 - iSER logouts cause kernel panic in 3.13 - Fix committed.

--

2) DHCP supporting IB properly:

- LP: #1401141 - ISC DHCP server does not work with InfiniBand - Fix
released.

--

3) QPID compilation problem:

- LP: #1367255 - qpid-cpp depends on ruby1.8 and should depend on
ruby1.9 - Will attach fix.

--

4) Libibverbs and Libmlx4 - Still checking ABI (if it was broken or not). If a fix is possible it will be made here.

All libibverbs and libmlx4 rdepends are safe and won't be touched:

- srptools
- rdmacm-utils
- perftest
- librdmawrap2
- libopenmpi1.6
- libnes1
- libmthca1
- libmlx5-1
- libipathverbs1
- libibcm1
- libcxgb3-1
- ibverbs-utils
- glusterfs-common
- fio
- libtotem-pg5
- librdmacm1

Packages libiverbs and libmlx4 do not break ABI compatibility thus rdepends do not need to be recompiled.

"""
The abi_compat field of struct ibv_context is used to determine
support of verbs extensions. As a result, support for ABI version < 2
is removed (corresponds to kernel releases 2.6.11-2.6.14 no longer
being supported). The lowest ABI now supported is 3 (really 4 since
 2.6.15 was ABI 4, I don't see that ABI 3 was in a release).
"""

Included patch called "Infrastructure-to-support-verbs-extensions.patch" only makes libibverbs not to support older kernels (ABI <= 2) which is acceptable for Trusty's 3.13 kernels.

Attaching fixes for libibverbs and libmlx4 soon...

libmlx4 does need specific version from libibverbs because it is also being changed to support:

- Flow steering control
- Offload support

thus this version, 1.0.5-1ubuntu2 has to be compiled exactly with libibverbs-dev (>= 1.1.7-1ubuntu2) and depends exactly on libibverbs1 (>= 1.1.7-1ubuntu2). There is no way to avoid that.

Attaching fixes for libibverbs and libmlx4.

The verification of the Stable Release Update for tgt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package libibverbs - 1.1.8-1ubuntu1.1

---------------
libibverbs (1.1.8-1ubuntu1.1) vivid; urgency=medium

  * Add checksum offload support capability (LP: #1409904)

 -- Rafael David Tinoco (Inaddy) <email address hidden> Tue, 21 Jul 2015 08:10:03 -0500

This bug was fixed in the package libmlx4 - 1.0.6-1ubuntu0.1

---------------
libmlx4 (1.0.6-1ubuntu0.1) vivid; urgency=medium

  * Add checksum offload support capability (LP: #1409904)

 -- Rafael David Tinoco (Inaddy) <email address hidden> Tue, 21 Jul 2015 08:31:10 -0500

This bug was fixed in the package tgt - 1:1.0.43-0ubuntu4.1

---------------
tgt (1:1.0.43-0ubuntu4.1) vivid; urgency=medium

  * iSER: Add Discovery support (LP: #1409904)

 -- Rafael David Tinoco (Inaddy) <email address hidden> Tue, 21 Jul 2015 08:02:51 -0500

Uploaded new tgt, libmlx4, and libibverbs updates to trusty. Will remove ubuntu-sponsors, since there aren't any more patches left.

Hello Brian, or anyone else affected,

Accepted libmlx4 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libmlx4/1.0.5-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Brian, or anyone else affected,

Accepted libibverbs into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libibverbs/1.1.7-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Brian, or anyone else affected,

Accepted tgt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tgt/1:1.0.43-0ubuntu4.1~14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Verified TGT iSER support:

Linux iserserver.main 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Linux iserclient.main 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

### SERVER

root@iserserver:~# dd if=/dev/zero of=/disks/lunit1 bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 10.3035 s, 104 MB/s
root@iserserver:~# tgt-setup-lun -n lunit1 -d /disks/lunit1 -t iser
Using transport: iser
Creating new target (name=iqn.2001-04.com.iserserver-lunit1, tid=1)
Adding a logical unit (/disks/lunit1) to target, tid=1
Accepting connections from all initiators
root@iserserver:~# ifconfig ib0
ib0 Link encap:UNSPEC HWaddr 80-00-05-8A-FE-80-00-00-00-00-00-00-00-00-00-00
          inet addr:172.16.0.21 Bcast:172.16.0.127 Mask:255.255.255.128

### CLIENT

inaddy@iserclient:~$ sudo apt-get install open-iscsi
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  open-iscsi
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 268 kB of archives.
After this operation, 2,221 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main open-iscsi amd64 2.0.873-3ubuntu9 [268 kB]
Fetched 268 kB in 0s (466 kB/s)
Selecting previously unselected package open-iscsi.
(Reading database ... 80686 files and directories currently installed.)
Preparing to unpack .../open-iscsi_2.0.873-3ubuntu9_amd64.deb ...
Unpacking open-iscsi (2.0.873-3ubuntu9) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up open-iscsi (2.0.873-3ubuntu9) ...
Processing triggers for ureadahead (0.100.0-16) ...
inaddy@iserclient:~$ sudo su -
root@iserclient:~# update-rc.d -f open-iscsi remove
 Removing any system startup links for /etc/init.d/open-iscsi ...
   /etc/rc0.d/K81open-iscsi
   /etc/rc1.d/K81open-iscsi
   /etc/rc6.d/K81open-iscsi
   /etc/rcS.d/S45open-iscsi
root@iserclient:~# service open-iscsi stop
 * Unmounting iscsi-backed filesystems [ OK ]
 * Disconnecting iSCSI targets [ OK ]
 * Stopping iSCSI initiator service [ OK ]
root@iserclient:~# service open-iscsi start
 * Starting iSCSI initiator service iscsid [ OK ]
 * Setting up iSCSI targets
iscsiadm: No records found
                                                                                  [ OK ]
 * Mounting network filesystems [ OK ]
root@iserclient:~# ifconfig bi0
bi0: error fetching interface information: Device not found
root@iserclient:~# ifconfig ib0
ib0 Link encap:UNSPEC HWaddr 80-00-05-8B-FE-80-00-00-00-00-00-00-00-00-00-00
          inet addr:172.16.0.22 Bcast:172.16.0.127 Mask:255.255.255.128
          inet6 addr: fe80::214:500:0:85/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
          RX packets:3960620 e...

Kamal, Pankaj,

----

I’m glad to inform that finally bug LP: #1409904 is being finalised (for Trusty) and its fix is already committed.

Next step is for us to “verify” the fix (from -proposed repository) and mark the public bug as “verified”.

I have already verified TGT iSER support:

https://bugs.launchpad.net/ubuntu/+source/libibverbs/+bug/1409904
https://bugs.launchpad.net/ubuntu/+source/libibverbs/+bug/1409904/comments/88

And also the iSER logout kernel panic bug (to be released soon):

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1443648

(Using a kernel >= 3.13.0-62-generic is enough)

----

The only thing I need from Mellanox is to verify packages “libibverbs” and “libmlx4” from -proposed.

I need Mellanox to have a Trusty installation and enable -proposed repository:

# /etc/apt/sources.list

deb http://archive.ubuntu.com/ubuntu/ trusty main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ trusty-proposed main restricted universe multiverse

$ sudo apt-get update && sudo apt-get install libibverbs1 libmlx4-1

And verify:

- Flow steering control
- Checksum offload

from both packages: libibverbs and libmlx4.

Please “reply-all” to this email right after verifying the fix. Do have in mind that the fix won’t be released until we provide feedback. Right after you verify the fix, if you don’t find any problem, I’ll change the public bug flag to “verification-done” and fix will be released.

Thank you very much, best regards….

Rafael Tinoco

Hello Pankaj,

I’m sorry for not being more clear…

There is no way that a SRU is accepted if -proposed package is not verified.

TGT iSER support was already verified by me.

I still need Kamal to verify libibverbs and libmlx4.

Thank you!

In a meeting with Mellanox in Israel I was told that libibverbs and libmlx4 have been verified and proved to work. Changing this to verification done. Thank you.

This bug was fixed in the package libmlx4 - 1.0.5-1ubuntu1.1

---------------
libmlx4 (1.0.5-1ubuntu1.1) trusty; urgency=medium

  * Remove compatibility with older libibverbs (LP: #1409904)
  * Update to use libibverbs extensions support (LP: #1409904)
  * Add receive flow steering support (LP: #1409904)
  * Add checksum offload support capability (LP: #1409904)

 -- Rafael David Tinoco <email address hidden> Wed, 05 Aug 2015 16:27:04 -0300

This bug was fixed in the package libibverbs - 1.1.7-1ubuntu1.1

---------------
libibverbs (1.1.7-1ubuntu1.1) trusty; urgency=medium

  * Add general defitions to support uverbs extensions (LP: #1409904)
  * Add receive flow steering support (LP: #1409904)
  * Add checksum offload support capability (LP: #1409904)

 -- Rafael David Tinoco <email address hidden> Wed, 05 Aug 2015 16:14:02 -0300

This bug was fixed in the package tgt - 1:1.0.43-0ubuntu4.1~14.04.1

---------------
tgt (1:1.0.43-0ubuntu4.1~14.04.1) trusty; urgency=medium

  * iSER: Add Discovery support (LP: #1409904)

 -- Rafael David Tinoco <email address hidden> Wed, 05 Aug 2015 17:21:17 -0500