libvirt raise Could not access KVM kernel module error

Bug #1715356 reported by Jeffrey Zhang
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Undecided
Unassigned
Pike
Triaged
Undecided
Unassigned

Bug Description

When deploying on arm64, permission denied reported:
Terminating instance: libvirtError: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied

Also hitting this on tripleo x86_64 deployments since https://review.openstack.org/492780 merged

Changed in kolla:
milestone: none → queens-1
no longer affects: kolla/pike
Changed in kolla:
milestone: none → queens-1
Changed in kolla:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/pike)

Reviewed: https://review.openstack.org/501909
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=b0e26030583f881934a42e6c5f121736f9f5341e
Submitter: Jenkins
Branch: stable/pike

commit b0e26030583f881934a42e6c5f121736f9f5341e
Author: Jeffrey Zhang <email address hidden>
Date: Wed Sep 6 17:59:39 2017 +0800

    Revert "nova-libvirt: fix kvm permission issue"

    This reverts commit 43650d5837372449294a7a22e9c9167a92744e96.

    This fix do not work with CentOS x86_64. Since I haven't arm64 env, and
    we have fix this soon before pike is release. Just revert this and fix
    in the future.

    Change-Id: Id9eb531de7d05051f38e3ed13b64ae7abf552767
    Partial-Bug: #1715356
    (cherry picked from commit e9929dae60033fda9426671628b8c874dde97480)

tags: added: in-stable-pike
Revision history for this message
Oliver Walsh (owalsh) wrote :

This needs to be mode 666 - there is a udev rule that does this:

[heat-admin@overcloud-novacomputebm-0 ~]$ rpm -qf /lib/udev/rules.d/80-kvm.rules
qemu-kvm-ev-2.6.0-28.el7.10.1.x86_64
[heat-admin@overcloud-novacomputebm-0 ~]$ cat /lib/udev/rules.d/80-kvm.rules
KERNEL=="kvm", GROUP="kvm", MODE="0666"

description: updated
Revision history for this message
Oliver Walsh (owalsh) wrote :

For reference this is the correct behaviour on Fedora/CentOS/RHEL: https://bugzilla.redhat.com/show_bug.cgi?id=497341

Also it's being moved to systemd from the qemu rpm, so I think we still need to mirror the behaviour in kolla - https://bugzilla.redhat.com/show_bug.cgi?id=1431876, which also mentions that Debian sets it to 0660 + group kvm.

Oliver Walsh (owalsh)
description: updated
Revision history for this message
Steven Dake (sdake) wrote :

Disagree on 666. Just because udev does it, doesn't mean it is correct. E.G. dd if=/dev/zero of=/dev/kvm will blow up the system. kvm should only be accessible by the qemu group. It is possible to properly set permissions for qemu:

e.g.:
https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/templates/qemu.conf.j2#L10-L11

Revision history for this message
Oliver Walsh (owalsh) wrote :

Clearly there isn't consensus on what it correct. The Fedora Virtualization Maintainers decided 0666, the Debian maintainers decided 0660. I think the respective mailing lists and/or bug trackers is a more appropriate place to discuss it.

I don't agree with kolla being opinionated on matters such as this. If it diverges from the expected behaviour it introduces subtle and confusing bugs, such as this.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/501653
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=f1b98a4925826333cf22dafe15ef755066c51d48
Submitter: Zuul
Branch: master

commit f1b98a4925826333cf22dafe15ef755066c51d48
Author: Kevin Zhao <email address hidden>
Date: Thu Sep 7 17:28:29 2017 +0800

    nova-libvirt: fix kvm permission issue

    When deploy in AArch64, will meet kvm permission issue.
    Fix it with Arch specified.

    Closes-bug: #1715356

    Change-Id: I09dc27693a83dc77704773a25547725de480dbe8
    Signed-off-by: Kevin Zhao <email address hidden>

Changed in kolla:
status: Triaged → Fix Released
Revision history for this message
xinliang (xin3liang) wrote :

The root cause for this issue is that qemu not run with nova user on debian.
If we set qemu running with nova user with this patch: https://review.openstack.org/#/c/525891/
Then we need to revert commit f1b98a4925826333cf22dafe15ef755066c51d48.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.openstack.org/525900

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 6.0.0.0b2

This issue was fixed in the openstack/kolla 6.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/525900
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=14cda91c4ef7020ac43f430869798387de15dae4
Submitter: Zuul
Branch: master

commit 14cda91c4ef7020ac43f430869798387de15dae4
Author: Xinliang Liu <email address hidden>
Date: Wed Dec 6 14:23:58 2017 +0800

    Revert "nova-libvirt: fix kvm permission issue"

    This is not needed. If we make sure qemu use nova user.
    Because nova user in group qemu.

    See comment #7 of bug #1715356:
    The root cause for this issue is that qemu not run with nova user on
    debian.
    If we set qemu running with nova user with this patch:
    https://review.openstack.org/#/c/525891/
    Then we need to revert commit f1b98a4925826333cf22dafe15ef755066c51d48.

    This reverts commit f1b98a4925826333cf22dafe15ef755066c51d48.
    Closes-bug: #1715356
    Depends-on: I36720af0c7d3dd7c69d2404843f54c0991aea1bb

    Change-Id: I62fbcf9e4ee5c3170c96576698f4ae8b66db1b74

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 6.0.0.0b3

This issue was fixed in the openstack/kolla 6.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.