NFS share not mounted due to findmnt behaviour change

Bug #1656774 reported by Lee Yarwood
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Lee Yarwood
Newton
In Progress
Undecided
Lee Yarwood

Bug Description

Description
===========
The following findmnt behaviour change present in util-linux-2.23.2-33 causes libvirt_utils.is_mounted to incorrectly return True when a share is already mounted on a host but not by Nova, for example in an allinone/devstack environment where cinder-volume is running on the same host and has already mounted the share :

findmnt --target behaviour changed in 7.3, shows all mount-points in chroot
https://bugzilla.redhat.com/show_bug.cgi?id=1405238

Steps to reproduce
==================
- Install devstack with the NFS plugin on a RHEL 7.3, CentOS 7.3 or Fedora 25 host.
- Attempt to launch a volume backed instance.

Expected result
===============
NFS share mounted by both Nova and Cinder allowing the instance to launch correctly.

Actual result
=============
NFS share only mounted by Cinder causing the instance to fail to launch.

Environment
===========
1. Exact version of OpenStack you are running. See the following
   list for all releases: http://docs.openstack.org/releases/

   devstack (master) + devstack-plugin-nfs (master) + CentOS 7.3

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   NFS

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   n/a

Logs & Configs
==============

Lee Yarwood (lyarwood)
Changed in nova:
importance: Undecided → High
Changed in nova:
assignee: nobody → Lee Yarwood (lyarwood)
status: New → In Progress
Changed in nova:
assignee: Lee Yarwood (lyarwood) → Matthew Booth (mbooth-9)
Lee Yarwood (lyarwood)
Changed in nova:
assignee: Matthew Booth (mbooth-9) → Lee Yarwood (lyarwood)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/420146
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=86b361d9166200db2174be87214190a29e77f98f
Submitter: Jenkins
Branch: master

commit 86b361d9166200db2174be87214190a29e77f98f
Author: Lee Yarwood <email address hidden>
Date: Fri Jan 13 16:45:32 2017 +0000

    libvirt: workaround findmnt behaviour change

    findmnt as provided by util-linux-2.23.2-33 now returns 0 when --source
    is already mounted on the host under a different mountpoint [1]. This causes
    issues in environments where cinder-volume may have already mounted a
    given NFS share.

    This change works around this behaviour change by using os.path.ismount
    to confirm if the target is already a mountpoint and if a source is
    provided also checking /proc/mounts to confirm that it is mounted at the
    target location.

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1405238

    Closes-Bug: #1656774
    Change-Id: Id64cc77237bf770a9a75bbf666a54a604ace558c

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/428743

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.0.0rc1

This issue was fixed in the openstack/nova 15.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/newton)

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/428743

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.