nvmeof connector _get_host_uuid incompatible with btrfs snapshots

Bug #2026257 reported by Orlando Chamberlain
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
os-brick
In Progress
Medium
Eric Harney
Ubuntu
Confirmed
Undecided
Unassigned

Bug Description

initiator/connectors/nvmeof.py _get_host_uuid uses `findmnt / -n -o SOURCE` to get the path of the block device to use with `blkid`. For the server I'm working on we have btrfs snapshots with snapper and that makes this not work.

I'm using openstack 2023.1 installed with devstack on Ubuntu Server 2022.04.

$ findmnt / -n -o SOURCE
/dev/sda2[/.snapshots/3/snapshot]

Adding the "-v" flag might help?

$ findmnt / -n -o SOURCE -v
/dev/sda2

I moved the findmnt binary to findmnt.bak and made this shell script:

$ cat /usr/bin/findmnt
#!/bin/bash
/usr/bin/findmnt.bak $@ -v

And now the python backtrace below doesn't happen when I try to live migrate a VM to this server:

DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=52748) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}}
DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=52748) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}}
DEBUG oslo.privsep.daemon [-] privsep: reply[e2abaa37-8e82-46e4-83ea-da43dfb24608]: (4, ('/dev/sdc2[/.snapshots/3/snapshot]\n', '')) {{(pid=52748) _call_back /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:501}}
DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sdc2[/.snapshots/3/snapshot] -s UUID -o value {{(pid=52748) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:384}}
DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sdc2[/.snapshots/3/snapshot] -s UUID -o value" returned: 2 in 0.007s {{(pid=52748) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:422}}
DEBUG oslo_concurrency.processutils [-] 'blkid /dev/sdc2[/.snapshots/3/snapshot] -s UUID -o value' failed. Not Retrying. {{(pid=52748) execute /usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py:473}}
DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a7caad1d-6c58-4372-a947-6e19bcaa109c]: Unexpected error while running command.
Command: blkid /dev/sdc2[/.snapshots/3/snapshot] -s UUID -o value
Exit code: 2
Stdout: ''
Stderr: '' {{(pid=52748) _process_cmd /usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py:479}}
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
    ret = func(*f_args, **f_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
    return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
    return putils.execute(on_execute=on_execute,
  File "/usr/local/lib/python3.10/dist-packages/oslo_concurrency/processutils.py", line 438, in execute
    raise ProcessExecutionError(exit_code=_returncode,
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: blkid /dev/sdc2[/.snapshots/3/snapshot] -s UUID -o value
Exit code: 2
Stdout: ''
Stderr: ''

Tags: nvmeof
Revision history for this message
Orlando Chamberlain (redecorating) wrote :

I've started using the following sed command on each server for now as a simpler fix:

sudo sed -i "s|'findmnt', '/', '-n', '-o', 'SOURCE'|'findmnt', '/', '-n', '-o', 'SOURCE', '-v'|" /usr/local/lib/python3.10/dist-packages/os_brick/initiator/connectors/nvmeof.py

This means I don't need to replace the findmnt binary anymore.

tags: added: nvmeof
Changed in os-brick:
importance: Undecided → Medium
Changed in os-brick:
status: New → Triaged
status: Triaged → New
Eric Harney (eharney)
Changed in os-brick:
assignee: nobody → Eric Harney (eharney)
summary: - _get_host_uuid incompatible with btrfs snapshots
+ nvmeof connector _get_host_uuid incompatible with btrfs snapshots
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu:
status: New → Confirmed
Revision history for this message
Eric Harney (eharney) wrote :
Changed in os-brick:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-brick 6.4.0

This issue was fixed in the openstack/os-brick 6.4.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.