[Hyper-V] hv: vmbus: Fix a host signaling bug

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1519897

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

No longs required, this is a request to pick up an upstream submission.

It looks like there was a V2 to the patch mentioned in the description:

https://lkml.org/lkml/2015/11/19/795

I'll build the test kernels using the V2 patch.

Yeah, at the time I submitted we had determined that the first patch would work, but that there might be dialogue in the upstreaming. The V2 submission is okay with us.

Dear Joshua,

I built a Xenial test kernel with the V2 patch.

Please test the kernel and let us know your results. If we can receive positive test feedback by December 18, we can proceed to submit the patches for kernel SRU review in an attempt to make the upcoming kernel SRU cycle starting on December 20th. Assuming the patches are accepted for SRU and land in the upcoming SRU cycle, the fix should be available from the -updates pocket by January 9th.

The Xenial test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1519897/xenial

This kernel will cover both Xenial and Wily testing. Vivid and Trusty will need some backporting of the patch, so separate test kernels will be posted for them.

Ah, we hadn't started Xenial testing yet. Give me a little more time to spin that up.

The patch is actually already in Vivid due to bug 1508706 .

Indeed. Sorry, I should have recognized it. Too many parallel threads of effort.

Also, it looks like it's the V1 version of the patch in Vivid. However, it will be replace by V2 when it comes down from stable as an update.

We have customers encountering this issue on 12.04.5. I have advised them to be sure they are running lts-trusty. Is there an outlook on when this fix may be available for that release?

The patch is in the mainline tree as the following commit:

8599846 - Drivers: hv: vmbus: Fix a Host signaling bug

However, it has not been tagged in an official release upstream as of yet. It was cc'd to stable, so it will make it's way into the stable kernels through the usual stable update process.

A SAUCE patch is in Vivid for this patch. I have a SAUCE patch for Wily and Xenial until the fix comes down from upstream. I can SRU the SAUCE patch once it is tested per comment #5.

The patch was not applying cleanly to Trusty and the lts-trusty kernels. I'll work on backporting that again and post a Trusty test kernel shortly.

Some of the prereqs needed for Trusty and 12.04.5 would come in via bug 1454892. We are just waiting for testing on that bug. However, I can still build a Trusty test kernel with the assumption that the prereqs will come in via bug 1454892.

I built a Trusty test kernel with commit 8599846 and all the necessary prereqs.

Please test the following kernel and let us know your results. If we can receive positive test feedback by January 29th, we can proceed to submit the patches for kernel SRU review in an attempt to make the upcoming kernel SRU cycle starting on January 29th. Assuming the patches are accepted for SRU and land in the upcoming SRU cycle, the fix should be available from the -updates pocket by February 20th.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1519897/trusty

Hi Joe,
This kernel looks good now, I was able to run the net perf against it and also the network tests and it looks good.
Please proceed with the SRU for this.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Do *not* merge these changes.
We're seeing a large network performance drop with this kernel, so we need to further investigate.

What other netvsc related patches have been included in the proposed kernel? Compared to the test kernel from comment #13.
The proposed kernel build tag is 3.13.0-78.122

From the ubuntu-trusty git repository:

$ git --format=oneline Ubuntu-3.13.0-77.121.. -- drivers/hv

04d1fd1012796e714f3b291fc0487e45d2fbd9ba Drivers: hv: vmbus: Fix a Host signaling bug
952f5d0a5b0f3b032902537a1aeafdee75ac69a6 Drivers: hv: vmbus: Fix a siganlling host signalling issue
c402db4908545dbe6daa1aa7494be37d6b3f4ee3 Drivers: hv: vmbus: Export the vmbus_sendpacket_pagebuffer_ctl()
14df727b252b7f20ffb156edab89d81ebb2b31b5 Drivers: hv: vmbus: Suport an API to send packet with additional control
cb5c01b81dc4b08c9eccd40b2f626a3c7bced6f5 Drivers: hv: vmbus: Suport an API to send pagebuffers with additional control
ee8c7cb858c2e7f2d81d9b87fdfb6ee292b341a9 Drivers: hv: vmbus: Implement per-CPU mapping of relid to channel
a84dc0bfcd6b8d9a217e7c73e89b1e3ed44ac3f8 Drivers: hv: Eliminate the channel spinlock in the callback path

These were the commits applied in the test kernel posted in comment #13. The SHA1s are different because this is the tree used to build my test kernel.

e8aae57 Drivers: hv: vmbus: Fix a Host signaling bug
5d9beeb Drivers: hv: vmbus: Fix a siganlling host signalling issue
4763d20 Drivers: hv: vmbus: Export the vmbus_sendpacket_pagebuffer_ctl()
eab3ca4 Drivers: hv: vmbus: Suport an API to send packet with additional control
be7bca0 Drivers: hv: vmbus: Suport an API to send pagebuffers with additional control
6d43771 Drivers: hv: vmbus: Implement per-CPU mapping of relid to channel
b09952c Drivers: hv: Eliminate the channel spinlock in the callback path

So the changes to drivers/hv in Trusty -proposed are the same. I can build a Trusty -proposed test kernel without these 7 patches to see if they are the cause. If they are not, there may be a new regression in this -proposed kernel that we need to figure out.

I built a Trust -proposed test kernel with the 7 commits reverted. Can you test this kernel and see if it still exhibits the performance regression? The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1519897/trusty-proposed/

Upstream commit 8599846d upstream should have put this into Wily/Xenial. Is that in there? If so, we can close those.

This fix has been applied to Wily and is in the 4.2.0-29.34 version that is currently in -proposed. Would be nice to get that tested to verify.

In response to comment #18, this probably needs http://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=757647e10e55c01fb7a9c4356529442e316a7c72

Neither my test kernel from comment #13 nor the -proposed Trusty kernel have commit d34af82 applied. However, my test kernel did not exhibit the performance regression. My test kernel had good results per comment #14. @Chris, could there have been an issue with the test and maybe my kernel did also show the performance issue?

Actually I was referring to commit 757647e10e in comment #23, which actually fixes a performance introduced by commit d34af82.

Brad, in regards to comment #21.

We've tested the proposed kernel v4.2.0-29.34 on Wily and the performance is looking good. Please merge the changes.

I have a small question in regards to versioning.
When doing the install of the proposed kernel, in one place in apt-get I get that "Selected version '4.2.0.29.31' (Ubuntu:15.10/wily-proposed [amd64]) for 'linux-image-generic'" however the actual package name is indeed showing as 4.2.0-29.34. The candidate kernel when running "apt-cache policy linux-image-generic | grep "Candidate"" for eg. is also showing up at 4.2.0.29.31, instead it will install the .34 package.
From where does this difference in the versions comes from?

Joe, we would like to also get your input on the below.
In the trusty kernel (Ubuntu-3.13.0-77.121 - https://code.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/trusty/+ref/master) we tried to include the below patch to see if that resolves the network performance regression issue.
Patch is http://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=757647e10e55c01fb7a9c4356529442e316a7c72

However, with the patch applied directly to the cloned trusty branch, the compilation fails with a lot of undeclared symbols.
Are you able to build a test kernel including the patches from comment #18 and with the above mentioned one?
Thank you!

@Chris, I'll investigate and get a test kernel build for you. It may be that commit 757647e requires some prereq commits.

Also, have you had a chance to test the kernel posted in comment #19 to see if the commits applied to the master-next tree were what introduced the performance regression?

The test kernel from comment #19 also has the network performance regression.

For that reason we are looking to test this with all the mentioned 7 patches and in addition with the patch from comment #26

The test kernel in comment #13 was built on top of Ubuntu-3.13.0-76.120. It had the 7 commits listed in comment #18 applied. It did not have the performance regression per comment #14.

The test kernel in comment #19 was built on top of Ubuntu-3.13.0-78.122 I had the 7 commits listed in comment #19 reverted, but it also had the 3.13.11-ckt34 stable updates applied. It still had the regression per comment #28.

This may indicated the regression came in via one of the upstream stable updates.

I'd like to perform a kernel bisect between Ubuntu-3.13.0-76.120 and Ubuntu-3.13.0-78.122. However, we first need to make sure Ubuntu-3.13.0-76.120 does not have the performance regression and Ubuntu-3.13.0-78.122 does have the regression.

Can you test these two kernels for confirmation. Once we have the results, we can start the bisect or decide on the next step:

Ubuntu-3.13.0-76.120: https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/ppa/+build/8865557

Ubuntu-3.13.0-78.122: https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/8926054

Note, you need to install both the linux-image and linux-image-extra .deb packages.

In parallel, I will still build you a test kernel with the original 7 commits and commit 757647e applied. I'm in the process of identifying all the prereqs and backporting.

Thanks Joe for the quick work on this.
To clarify on the distro release to use, we tested all kernels from this thread on top of Trusty.
However, this kernel is supposed to go for Precise - 12.04.

Question is, if it's safe to continue do all the testing on Trusty, or actually switch to Precise? I don't see a difference in all this, but I want to be on the same page.

I think we should stick to testing on Trusty for now. The backport kernel goes into Trusty(As the -updates kernel) and then into Precise. Since we see this regression with Trusty as the userspace, we should continue to test there.

If you find that Ubuntu-3.13.0-76.120 is good and it does not cause the performance regression, can you test Ubuntu-3.13.0-77.121 as well? This will shorten our bisect window.

The Ubuntu-3.13.0-77.121 kernel can be downloaded from:
https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/8874360

To start the bisect, we want to be sure we identify the last good kernel and first bad one.

Based on the numbers here, I will continue to include some patches or suggestions we got from Haiyang, and send an update later.
You should be able to push the update as this seems to be a problem with 3.13 entire tree, then in a separate thread we can identify the missing patches.

These are the numbers that we have when running a simple test with ntttcp.
All the below 3 kernels have been installed from the trusty-proposed repo.

As a reference, on the same system but with the latest available kernel we have this:
Kernel: lts-vivid - 3.19.0.50 (56)
Throughput: 12.87Gbps

Our historical perf testing includes only kernel 3.16.0-59 for 14.04, for which we got a throughput of 18.02 Gbps.

Distribution release: Ubuntu 14.04.3

Kernel: 3.13.0-76-generic (120)
Throughput: 1.53Gbps

Kernel: 3.13.0-77-generic (121)
Throughput: 1.41Gbps

Kernel: 3.13.0-78-generic (122)
Throughput: 1.50Gbps

Going back to try and find a good build, picking up the first 3.13 kernel:
Kernel: 3.13.0-24-generic (122)
Throughput: 1.50Gbps

@chris,

As I tried to explain yesterday we can't release a kernel with known regressions in it. We need your help in bisecting the kernel to find the commit that is causing the regression. The testing we are doing in-house is not showing the regression.

@brad-figg We are very sorry.

The bottom line is that we are not seeing good performance on 3.13 going back for many builds, with or without the 7 patches we are requesting. Do not hold any release for resolution of this issue. I recommend going forward with the 7 patches in place.

The difference in performance is compared to our 14.04 baseline, (our 14.04.2 baseline) which was with a different series of kernel, 3.16. As you may know, we put a lot of effort into utopic (and vivid, and wily) to make the LTS as good as possible on Hyper-V.

There are multiple bugs in which we were unable to take things back to 3.13, and it's clear that something important is still missing. If there was some way for customer on 12.04 to run later HWE kernels, maybe that is the ultimate solution.

@Chris,

I'm reviewing comment #33. Just to confirm, you do not believe this performance issue is a regression with the 3.13.0-78.122 Trusty kernel in -proposed? All of the Trusty kernels going back to 3.13.0-24 have this issue?

If that is the case, this may be an entirely different bug that we want to address separately.

@chris,

Sorry for my confusion. If this is not a new regression then we are ok to release.

@josh

Thanks for the update. I'll continue to see if we can build a Trusty test kernel with commit 757647e applied.

@jsalisbury Thanks, I think that is the best path forward. The flow dissector will stop the bunching up of activity on CPU 0 which will help us see what other items are missing. 3.16 is better, even without it, though. Vivid 3.19 has a form of it now. Wily 4.2 also has a form of it. Xenial as well. The upstream patch just uses a different hash than we do, but the difference is minimal.

Between this and the NUMA set we should be able to see what it needful.

This bug was fixed in the package linux - 3.13.0-79.123

---------------
linux (3.13.0-79.123) trusty; urgency=low

  [ Seth Forshee ]

  * SAUCE: cred: Add clone_cred() interface
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Use mounter's credentials instead of full kernel
    credentials
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.*
    xattrs
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Be more careful about copying up sxid files
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Propogate nosuid from lower and upper mounts
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576

linux (3.13.0-78.122) trusty; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1540559

  [ Eric Dumazet ]

  * SAUCE: (no-up) udp: properly support MSG_PEEK with truncated buffers
    - LP: #1527902

  [ J. R. Okajima ]

  * SAUCE: ubuntu: aufs: tiny, extract a new func xino_fwrite_wkq()
    - LP: #1533043
  * SAUCE: ubuntu: aufs: for 4.3, XINO handles EINTR from the dying process
    - LP: #1533043

  [ Upstream Kernel Changes ]

  * Revert "[stable-only] net: add length argument to
    skb_copy_and_csum_datagram_iovec"
    - LP: #1538756
  * unregister_netdevice : move RTM_DELLINK to until after ndo_uninit
    - LP: #1525324
  * rtnetlink: delay RTM_DELLINK notification until after ndo_uninit()
    - LP: #1525324
  * Drivers: hv: Eliminate the channel spinlock in the callback path
    - LP: #1519897
  * Drivers: hv: vmbus: Implement per-CPU mapping of relid to channel
    - LP: #1519897
  * Drivers: hv: vmbus: Suport an API to send pagebuffers with additional
    control
    - LP: #1519897
  * Drivers: hv: vmbus: Suport an API to send packet with additional
    control
    - LP: #1519897
  * Drivers: hv: vmbus: Export the vmbus_sendpacket_pagebuffer_ctl()
    - LP: #1519897
  * Drivers: hv: vmbus: Fix a siganlling host signalling issue
    - LP: #1519897
  * Drivers: hv: vmbus: Fix a Host signaling bug
    - LP: #1519897
  * ARC: Fix silly typo in MAINTAINERS file
    - LP: #1538756
  * ip6mr: call del_timer_sync() in ip6mr_free_table()
    - LP: #1538756
  * gre6: allow to update all parameters via rtnl
    - LP: #1538756
  * atl1c: Improve driver not to do order 4 GFP_ATOMIC allocation
    - LP: #1538756
  * sctp: use the same clock as if sock source timestamps were on
    - LP: #1538756
  * sctp: update the netstamp_needed counter when copying sockets
    - LP: #1538756
  * ipv6: sctp: clone options to avoid use after free
    - LP: #1538756
  * net: add validation for the socket syscall protocol argument
    - LP: #1538756
  * sh_eth: fix kernel oops in skb_put()
    - LP: #1538756
  * pptp: verify sockaddr_len in pptp_bind() and pptp_connect()
    - LP: #1538756
  * bluetooth: Validate socket address length in sco_sock_bind().
    - LP: #1538756
  * af_unix: Revert 'lock_interruptible' in stream receive code
    - LP: #1538756
  * KEYS: Fix race between read and revoke
    - LP: #1538756
  * tools: Add a "make all" rule
    - LP: #1538...

This bug was fixed in the package linux-lts-trusty - 3.13.0-79.123~precise1

---------------
linux-lts-trusty (3.13.0-79.123~precise1) precise; urgency=low

  [ Seth Forshee ]

  * SAUCE: cred: Add clone_cred() interface
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Use mounter's credentials instead of full kernel
    credentials
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Skip permission checking for trusted.overlayfs.*
    xattrs
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Be more careful about copying up sxid files
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576
  * SAUCE: overlayfs: Propogate nosuid from lower and upper mounts
    - LP: #1534961, #1535150
    - CVE-2016-1575 CVE-2016-1576

linux (3.13.0-78.122) trusty; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1540559

  [ Eric Dumazet ]

  * SAUCE: (no-up) udp: properly support MSG_PEEK with truncated buffers
    - LP: #1527902

  [ J. R. Okajima ]

  * SAUCE: ubuntu: aufs: tiny, extract a new func xino_fwrite_wkq()
    - LP: #1533043
  * SAUCE: ubuntu: aufs: for 4.3, XINO handles EINTR from the dying process
    - LP: #1533043

  [ Upstream Kernel Changes ]

  * Revert "[stable-only] net: add length argument to
    skb_copy_and_csum_datagram_iovec"
    - LP: #1538756
  * unregister_netdevice : move RTM_DELLINK to until after ndo_uninit
    - LP: #1525324
  * rtnetlink: delay RTM_DELLINK notification until after ndo_uninit()
    - LP: #1525324
  * Drivers: hv: Eliminate the channel spinlock in the callback path
    - LP: #1519897
  * Drivers: hv: vmbus: Implement per-CPU mapping of relid to channel
    - LP: #1519897
  * Drivers: hv: vmbus: Suport an API to send pagebuffers with additional
    control
    - LP: #1519897
  * Drivers: hv: vmbus: Suport an API to send packet with additional
    control
    - LP: #1519897
  * Drivers: hv: vmbus: Export the vmbus_sendpacket_pagebuffer_ctl()
    - LP: #1519897
  * Drivers: hv: vmbus: Fix a siganlling host signalling issue
    - LP: #1519897
  * Drivers: hv: vmbus: Fix a Host signaling bug
    - LP: #1519897
  * ARC: Fix silly typo in MAINTAINERS file
    - LP: #1538756
  * ip6mr: call del_timer_sync() in ip6mr_free_table()
    - LP: #1538756
  * gre6: allow to update all parameters via rtnl
    - LP: #1538756
  * atl1c: Improve driver not to do order 4 GFP_ATOMIC allocation
    - LP: #1538756
  * sctp: use the same clock as if sock source timestamps were on
    - LP: #1538756
  * sctp: update the netstamp_needed counter when copying sockets
    - LP: #1538756
  * ipv6: sctp: clone options to avoid use after free
    - LP: #1538756
  * net: add validation for the socket syscall protocol argument
    - LP: #1538756
  * sh_eth: fix kernel oops in skb_put()
    - LP: #1538756
  * pptp: verify sockaddr_len in pptp_bind() and pptp_connect()
    - LP: #1538756
  * bluetooth: Validate socket address length in sco_sock_bind().
    - LP: #1538756
  * af_unix: Revert 'lock_interruptible' in stream receive code
    - LP: #1538756
  * KEYS: Fix race between read and revoke
    - LP: #1538756
  * too...