Error mounting device on create instance (LXC)

Bug #1829506 reported by Miguel Ángel Herranz Trillo
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Miguel Ángel Herranz Trillo
Queens
Fix Committed
Medium
Miguel Ángel Herranz Trillo
Rocky
Fix Committed
Medium
Matt Riedemann
Stein
Fix Committed
Medium
Matt Riedemann

Bug Description

Description
===========

After installing Openstack in Ubuntu 18.04 and using LXC as virtualization engine, the creation of an instance fails with a log in `nova-compute.log` reporting that:

 ```Instance failed to spawn: TypeError: mount() takes exactly 4 arguments (3 given)```

Extra analysis of the logs an code points towards this file of `python-nova` package:

 /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py

The problem seems that the call at line 251

```
   246 def mnt_dev(self):
   247 """Mount the device into the file system."""
   248 LOG.debug("Mount %(dev)s on %(dir)s",
   249 {'dev': self.mapped_device, 'dir': self.mount_dir})
   250 out, err = nova.privsep.fs.mount(None, self.mapped_device,
   251 self.mount_dir)
   252 if err:
   253 self.error = _('Failed to mount filesystem: %s') % err
   254 LOG.debug(self.error)
   255 return False
   256
   257 self.mounted = True
   258 return True
```

is not matching the function signature defined in /usr/lib/python2.7/dist-packages/nova/privsep/fs.py as:

```
    30 def mount(fstype, device, mountpoint, options):
    31 mount_cmd = ['mount']
    32 if fstype:
    33 mount_cmd.extend(['-t', fstype])
    34 if options is not None:
    35 mount_cmd.extend(options)
    36 mount_cmd.extend([device, mountpoint])
    37 return processutils.execute(*mount_cmd)
```

So, it should be safe just to add `None` as `options` parameter.

Steps to reproduce
==================

Environment info:

  OS: Ubuntu 18.04.2 LTS
  Kernel: Linux 4.15.0-46-generic #49-Ubuntu SMP x86_64
  Openstack: queens

  Related packages versions:

  nova-api: Installed: 2:17.0.9-0ubuntu1
  python-nova: Installed: 2:17.0.9-0ubuntu1
  nova-compute: Installed: 2:17.0.9-0ubuntu1
  nova-compute-lxc: Installed: 2:17.0.9-0ubuntu1

Command executed to create the instance:

```
# openstack server create --flavor lxc.small --image lxc_ubuntu_18.04 --nic net-id=${NEUTRON_NETWORK_NAT_ID} --security-group default --key-name vagrant test_lxc_instance
+-------------------------------------+---------------------------------------------------------+
| Field | Value |
+-------------------------------------+---------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | GhYewcoXch3w |
| config_drive | |
| created | 2019-05-17T10:00:57Z |
| flavor | lxc.small (0) |
| hostId | |
| id | fe69997d-10c2-4850-bdba-2468da7148d6 |
| image | lxc_ubuntu_18.04 (de3ca120-42f6-47a2-be1b-65c7276f1566) |
| key_name | vagrant |
| name | test_lxc_instance |
| progress | 0 |
| project_id | 4c2cf8fa37c8451d87ddd2cf28cfd6bc |
| properties | |
| security_groups | name='35a8788c-364b-4da3-9c26-4aebf0ac04c4' |
| status | BUILD |
| updated | 2019-05-17T10:00:57Z |
| user_id | 0d48615bb8c747f4b5ec5bfc693b6832 |
| volumes_attached | |
+-------------------------------------+---------------------------------------------------------+
```

Expected result
===============
```
# openstack server list
+--------------------------------------+-------------------+--------+-----------------+------------------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------------+--------+-----------------+------------------+-----------+
| dac7b2b8-841b-4a16-a54f-2d5177f05934 | test_lxc_instance | ACTIVE | nat=10.123.1.11 | lxc_ubuntu_18.04 | lxc.small |
+--------------------------------------+-------------------+--------+-----------------+------------------+-----------+
```

Actual result
=============
```
# openstack server list
+--------------------------------------+-------------------+--------+----------+------------------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------------+--------+----------+------------------+-----------+
| fe69997d-10c2-4850-bdba-2468da7148d6 | test_lxc_instance | ERROR | | lxc_ubuntu_18.04 | lxc.small |
+--------------------------------------+-------------------+--------+----------+------------------+-----------+
```

Logs
====

Included some logs generated during the command, and retrieved using:

```
grep -r "2019-05-17 10:0[01]" /var/log >> openstack_queens_lxc_instance_creation_mount_error.logs
```

Revision history for this message
Miguel Ángel Herranz Trillo (maherranzt) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/659780

Changed in nova:
assignee: nobody → Miguel Ángel Herranz Trillo (maherranzt)
status: New → In Progress
Matt Riedemann (mriedem)
tags: added: libvirt lxc privsep
Revision history for this message
Matt Riedemann (mriedem) wrote :

This is a regression since Queens:

https://review.opendev.org/#/c/494423/

I'm marking it as medium priority but it could probably just as well be low priority given I haven't heard of someone trying to use the libvirt+lxc configuration in years.

Changed in nova:
importance: Undecided → Medium
Changed in nova:
assignee: Miguel Ángel Herranz Trillo (maherranzt) → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/659780
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d2ef1ce309c28a5416af4cc00662ad6925574004
Submitter: Zuul
Branch: master

commit d2ef1ce309c28a5416af4cc00662ad6925574004
Author: Miguel Herranz <email address hidden>
Date: Fri May 17 12:53:10 2019 +0200

    Fix type error on call to mount device

    The call in nova.virt.disk.mount.api.Mount.mnt_dev() to
    nova.privsep.fs.mount() should include the `options` argument to
    fulfill with the method signature.

    The test test_do_mount_need_to_specify_fs_type has been modified to
    check that the caller use the correct signature.

    Closes-Bug: 1829506

    Change-Id: Id14993db6ea33b2da14caa4b58671fc57c182706
    Signed-off-by: Miguel Herranz <email address hidden>

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/667921

Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Miguel Ángel Herranz Trillo (maherranzt)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/stein)

Reviewed: https://review.opendev.org/667921
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8371073ac7a3439ea5b6ce9091a91e77cf88020e
Submitter: Zuul
Branch: stable/stein

commit 8371073ac7a3439ea5b6ce9091a91e77cf88020e
Author: Miguel Herranz <email address hidden>
Date: Fri May 17 12:53:10 2019 +0200

    Fix type error on call to mount device

    The call in nova.virt.disk.mount.api.Mount.mnt_dev() to
    nova.privsep.fs.mount() should include the `options` argument to
    fulfill with the method signature.

    The test test_do_mount_need_to_specify_fs_type has been modified to
    check that the caller use the correct signature.

    Closes-Bug: 1829506

    Change-Id: Id14993db6ea33b2da14caa4b58671fc57c182706
    Signed-off-by: Miguel Herranz <email address hidden>
    (cherry picked from commit d2ef1ce309c28a5416af4cc00662ad6925574004)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/669629

Revision history for this message
Miguel Ángel Herranz Trillo (maherranzt) wrote :

Cherry picked from master branch to stable/queens branch: https://review.opendev.org/#/c/669629/1

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/669664

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.opendev.org/669664
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e3cd1d9baa74181cbac1765291ad67654b35721d
Submitter: Zuul
Branch: stable/rocky

commit e3cd1d9baa74181cbac1765291ad67654b35721d
Author: Miguel Herranz <email address hidden>
Date: Fri May 17 12:53:10 2019 +0200

    Fix type error on call to mount device

    The call in nova.virt.disk.mount.api.Mount.mnt_dev() to
    nova.privsep.fs.mount() should include the `options` argument to
    fulfill with the method signature.

    The test test_do_mount_need_to_specify_fs_type has been modified to
    check that the caller use the correct signature.

    Closes-Bug: 1829506

    Change-Id: Id14993db6ea33b2da14caa4b58671fc57c182706
    Signed-off-by: Miguel Herranz <email address hidden>
    (cherry picked from commit d2ef1ce309c28a5416af4cc00662ad6925574004)
    (cherry picked from commit 8371073ac7a3439ea5b6ce9091a91e77cf88020e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.opendev.org/669629
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f77c628b4f08a625bebe8b8b3bb24a8e9784a226
Submitter: Zuul
Branch: stable/queens

commit f77c628b4f08a625bebe8b8b3bb24a8e9784a226
Author: Miguel Herranz <email address hidden>
Date: Fri May 17 12:53:10 2019 +0200

    Fix type error on call to mount device

    The call in nova.virt.disk.mount.api.Mount.mnt_dev() to
    nova.privsep.fs.mount() should include the `options` argument to
    fulfill with the method signature.

    The test test_do_mount_need_to_specify_fs_type has been modified to
    check that the caller use the correct signature.

    Closes-Bug: 1829506

    Change-Id: Id14993db6ea33b2da14caa4b58671fc57c182706
    Signed-off-by: Miguel Herranz <email address hidden>
    (cherry picked from commit d2ef1ce309c28a5416af4cc00662ad6925574004)
    (cherry picked from commit 8371073ac7a3439ea5b6ce9091a91e77cf88020e)
    (cherry picked from commit e3cd1d9baa74181cbac1765291ad67654b35721d)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.2

This issue was fixed in the openstack/nova 19.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.2.2

This issue was fixed in the openstack/nova 18.2.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.12

This issue was fixed in the openstack/nova 17.0.12 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 20.0.0.0rc1

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.