[Hyper-V] KVP daemon fails to run scripts

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1766857

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Did this bug just start happening with the 4.15.0-20 kernel? Were there prior 18.04 kernels that did not exhibit this bug?

I tested back to the daily build from 2nd of March and I can confirm the second issue (regarding the binaries path), but not the first one (regarding the daemon crash). On the build from the 2nd of March the linux-tools and linux-cloud-tools version is 4.15.0-10.11

Status changed to 'Confirmed' because the bug affects multiple users.

I confirm the second, the daemon hv_kvp_daemon not find the files:
/usr/libexec/hypervkvpd/hv_get_dns_info
/usr/libexec/hypervkvpd/hv_get_dhcp_info
With linux-cloud-tools version 4.15.0-22.24

In my own VMs, this does not affect linux-azure, it only affects the generic 4.15 kernel.

Commit 7fa32e5ec28b was mentioned in the descriptions. However, that commit has been in mainlien since 4.15-rc3, so it's always been in 18.04.

We may want to split the two issues in to two separate bugs, so they each get addressed.

Do both issues still happen with the latest Bionic kernel? Per comment #6, it sounds liek the second issues still happens. How about the first issue, where the KVP daemon crashes after approximatively 2 minutes of uptime?

Commit 7fa32e5ec28b was actually mentioned in bug 1664663, an I don't think it factors into this bug.

I have tested Bionic with kernel 4.15.0-22-generic, linux-cloud-tools-4.15.0-22.24 and linux-tools-4.15.0-22.24 and both issues still occur.

While testing the above mentioned kernel, I saw the following behavior:

1. Right after booting the system the KVP daemon reports it is active (running). The command "systemctl status hv-kvp-daemon" returns the following output:

hv-kvp-daemon.service - Hyper-V KVP Protocol Daemon
   Loaded: loaded (/lib/systemd/system/hv-kvp-daemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-06-07 11:43:22 UTC; 55s ago
 Main PID: 1363 (hv_kvp_daemon)
    Tasks: 1 (limit: 4496)
   CGroup: /system.slice/hv-kvp-daemon.service
           └─1363 /usr/lib/linux-tools/4.15.0-22-generic/hv_kvp_daemon -n

Jun 07 11:43:22 bionic systemd[1]: Started Hyper-V KVP Protocol Daemon.
Jun 07 11:43:22 bionic KVP[1363]: KVP starting; pid is:1363
Jun 07 11:43:22 bionic KVP[1363]: KVP LIC Version: 3.1

2. After approximately 2 minutes the KVP daemon enters in the failed state (issue 1).
3. Manually starting the daemon with the command "systemctl start hv-kvp-daemon" works perfectly fine.
4. "systemctl status hv-kvp-daemon" will now show the second issue:

hv-kvp-daemon.service - Hyper-V KVP Protocol Daemon
   Loaded: loaded (/lib/systemd/system/hv-kvp-daemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-06-07 11:47:29 UTC; 8s ago
 Main PID: 1995 (hv_kvp_daemon)
    Tasks: 1 (limit: 4496)
   CGroup: /system.slice/hv-kvp-daemon.service
           └─1995 /usr/lib/linux-tools/4.15.0-22-generic/hv_kvp_daemon -n

Jun 07 11:47:29 bionic systemd[1]: Started Hyper-V KVP Protocol Daemon.
Jun 07 11:47:29 bionic KVP[1995]: KVP starting; pid is:1995
Jun 07 11:47:29 bionic KVP[1995]: KVP LIC Version: 3.1
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found

5. Rebooting the system after this point will no longer trigger the first issue, only the second one. Even stopping the VM and turning it back ON does NOT trigger the first issue again.

Could it be that while the hv-kvp-daemon service shows the two files (hv_get_dns_info and hv_get_dhcp_info) are not found, it will not enter in the failed state? Meaning that the two apparently separate issues, are actually related. I am not 100% sure about this.

In order to trigger the first issue again, the Integration Service corresponding to the KVP daemon has to be disabled and re-enabled, and the VM has to be rebooted. This is just the way I managed to reproduce the first issue again, I am not sure if there are other ways to trigger it again.

Since they might be related, lets focus on the second issue first, which is cause by the missing files.

I'll set up a test system to try and reproduce.

It seems like this is a regression, but it would be good to confirm. Can you test these two kernels?

Last 4.14 Bionic kernel:
https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/ppa2/+build/14223014

First 4.15 Bionic kernel:
https://launchpad.net/ubuntu/+source/linux/4.15.0-9.10/+build/14326410

I built a test kernel which can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1766857/

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

Regarding comment #11:
I have tested both kernels provided with the following results:

- 4.14 Bionic kernel (4.14.0-16-generic) does NOT exhibit issue #2 (missing files), but it does exhibit issue #1 (KVP daemon failure after boot). Linux-cloud-tools version installed is 4.14.0-16.19.

- 4.15 Bionic kernel (4.15.0-9-generic) behaves the same as 4.14, meaning that issue #2 (missing files) is NOT present, but issue #1 (KVP daemon failure after boot) shows up. Linux-cloud-tools version is 4.15.0-9.10.

Regarding comment #12:
I have tested the kernel from the link and it seems that this kernel does NOT exhibit issue #2 (but it does exhibit issue #1).

At this point, we should probably have two separate bugs to address each problem. Can we keep this bug for the second issue and create a new bug for the first issue?

I have opened a different bug for the first issue: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1779640

Regarding the second issue, as mentioned in comment #13, the tested kernels did not exhibit issue #2.

Did you notice any other issues that occur when testing the test kernel posted in comment #12. The second issue posted in the bug is resolved by changing the definition of KVP_CONFIG_LOC in hv_kvp_daemon.c:

---
 tools/hv/hv_kvp_daemon.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
index 4c99c57..e969057 100644
--- a/tools/hv/hv_kvp_daemon.c
+++ b/tools/hv/hv_kvp_daemon.c
@@ -95,7 +95,7 @@ static struct utsname uts_buf;
 #define KVP_CONFIG_LOC "/var/lib/hyperv"

 #ifndef KVP_SCRIPTS_PATH
-#define KVP_SCRIPTS_PATH "/usr/libexec/hypervkvpd/"
+#define KVP_SCRIPTS_PATH "/usr/sbin/"
 #endif

 #define KVP_NET_DIR "/sys/class/net/"
--
2.7.4

Joe, for the 2nd issue initially reported here, about "After the KVP daemon is being started the following messages appear".
We've noticed a regression for this in 18.10 - daily's and even GA.

Can you please add Cosmic 18.10 to the bug here and re-open or should I send a new bug report for it?
Thank you!

Hi Chris, I added a Cosmic 18.10 bug task. Does this bug only happen in Cosmic, or is it still happening in Bionic?

Issue with 'Invalid kvp dependency scripts path /usr/libexec/hypervkvpd' remains in latest kernel-image bionic 4.15.0-38.41.

Verified azure-kernel 4.15.0-1030.31 for bionic, same VM and environment, and the issue does *not* appear.

Hmm, I wonder if the linux-azure kernel has and extra patch(s) that the generic kernel does not. I'll do some comparisons.

Just created a new VM and wondered why enhanced mode logins in hyper-V weren't working and encountered this.

$ uname -a
Linux cognite-lifeless 5.3.0-45-generic #37-Ubuntu SMP Thu Mar 26 20:41:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.10
Release: 19.10
Codename: eoan

Can confirm the Azure kernel also fixes enhanced sessions when combined with appropriate windows side metadata shenanigans.

Uups, wrong tagging. This bug does still affect linux (Ubuntu Focal)

I'm seeing the problem with /usr/libexec/hypervkvpd being used by hv_kvp_daemon, when the scripts are in /usr/sbin on a newly created Ubuntu 22.04 VM (on Windows 11 Pro 22H2 Hyper-V), on the linux-azure kernel.

murph@Ubuntu-Virtual-Machine:~$ uname -a
Linux Ubuntu-Virtual-Machine 5.15.0-1035-azure #42-Ubuntu SMP Tue Feb 28 19:41:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
murph@Ubuntu-Virtual-Machine:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy
murph@Ubuntu-Virtual-Machine:~$

There's an endless stream of log noise in syslog, at the fairly high rate of 8 logs per 10 seconds, with the same thing repeated each time:

Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35236]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35238]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35246]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35248]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35256]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35258]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35266]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35268]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found

Same problem with a newly created Ubuntu 20.04 VM (Windows 11 Pro 22H2 Hyper-V), on the linux-generic kernel. The same 8 logs per 10 seconds as seen on jammy, for the scripts not found in /usr/libexec/hyperkvpd.

murph@Ubuntu-Virtual-Machine:~$ uname -a
Linux Ubuntu-Virtual-Machine 5.4.0-146-generic #163-Ubuntu SMP Fri Mar 17 18:26:02 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
murph@Ubuntu-Virtual-Machine:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal
murph@Ubuntu-Virtual-Machine:~$

So, I've tried to dig into this a little to figure out why there's a mismatch between hv_kvp_daemon and the installed location of the hv_* scripts, and it looks like it must be somewhere on the packaging side of things rather than the build side. In both of the following Makefiles, the scripts which were throwing errors (hv_get_dhcp_info and hv_get_dns_info) look like they should be correctly installed in /usr/libexec/hypervkvpd. The daemons are still getting put into /usr/sbin by the Makefile, and they might be better living under libexec as well, since they don't really belong in PATH; but it's the location of the scripts which is a noisy bug that breaks some Hyper-V functionality.

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/tree/tools/hv/Makefile
https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-azure/+git/jammy/tree/tools/hv/Makefile

Something in the packaging phase is erroneously moving those scripts from /usr/libexec/hyperkvpd to /usr/sbin, but I'm not entirely sure just what is doing that, only that it doesn't seem to be on the build side.

With some more digging, I believe I've found the root of the bug.

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/tree/debian/rules.d/3-binary-indep.mk

(see also 2-binary-arch.mk for the hv_* daemon binaries, and how they get into their installed location)

That makefile is installing the scripts into $(cloudsbin), which looks like it defaults to /usr/sbin, contrary to the path embedded in the hv_kvp_daemon binary and the corresponding Makefile over in jammy/tools/hv.

I think hv_* should live in /usr/libexec/hypervkvpd, and only lsvmbus should live in /usr/sbin (as it has some diagnostic utility). I think rules.d/* needs to changed to eliminate the path mismatch involved in this bug. If consensus says these things should live in /usr/sbin, then I'm ok with that too (and the path embedded in the hv_kvp_daemon binary needs to be changed).

I've just checked the kinetic source, and the same path mismatch seems to exist between debian/rules.d/3-binary-indep.mk and tools/hv/*

Checked lunar source, looks like it has the same problem.

I've done a little more digging. The use of /usr/libexec/hypervkvpd for the child scripts comes from upstream.

https://github.com/torvalds/linux/blob/master/tools/hv/Makefile
https://github.com/torvalds/linux/blob/master/tools/hv/hv_kvp_daemon.c

If the governing principle is to adhere to the default Linux & Debian paths (where reasonable), then the child scripts (hv_*.sh) should live there in Ubuntu. I.e. this is a packaging error that should be fixed in Ubuntu's debian/rules.d/3-binary-indep.mk.

The default install location (in torvalds/linux) for the 3 hv_*_daemon binaries is /usr/sbin, but the hv_*.sh scripts install into /usr/libexec/hyperkvpd. For Debian, the hv_*_daemon binaries are in their hyperv-daemons package and install in /usr/sbin, but it seems that they have a long standing bug (#919350) where they don't provide any of the /usr/libexec/hyperkvpd scripts.