Nova 17.04 fails to create ephemeral storage with rbd driver: TypeError: name must be a string

So the disk_name has unicode in it? Looks like it should just be the instance uuid, so no unicode.

Looks like you might need this fix?

https://github.com/ceph/ceph/commit/72dc17ce327dc6422a2d62348b2c42971d364b6b

As a disk_name I had "4adc2422-b59d-4e84-b6fd-7f2e2f66d40a_disk", but when I tried to log type(disk_name) I've received <type 'unicode'>

It seems, that provided fix is for rbd library for python 3, while python 2.7 is used. So I don't have cstr() function in my rbd.py at all. Most likely, that the point is on version of rbd package and I should somehow upgrade it (and report the problem to OSA project, as it was installed by it):
~# pip show rbd
Name: rbd
Version: 2.0.0
Summary: Python bindings for the RBD library
Home-page: https://github.com/ceph/ceph/tree/master/src/pybind/rbd
Author: UNKNOWN
Author-email: UNKNOWN
License: LGPLv2+
Location: /usr/lib/python2.7/dist-packages
Requires:
~#

For example cinder in its cinder/volume/drivers/rbd.py resolves the problem like this:

 144 self.volume = driver.rbd.Image(rados_ioctx,
 145 utils.convert_str(name),
 146 snapshot=snapshot,
 147 read_only=read_only)

We had a similar bug around this awhile back [1].

Digging around in the code, I think I figured out why we're getting unicode type strings instead of native bytes strings in python2, it's because we're using this self.parse_url method to obtain the pool names:

https://github.com/openstack/nova/blob/5164517/nova/virt/libvirt/storage/rbd_utils.py#L180-L193

which calls six.moves.urllib.parse.unquote, which "Contains functions from Python 3’s urllib.parse...", which would be urllib.parse.unquote, which uses utf-8 encoding by default [3].

If you look throughout the rbd_utils.py file, there are many whack-a-mole calls of str() added to the various rbd library calls -- looks like we probably could have fixed this in one place in parse_url to have it return the a six.text_type, or something like that.

[1] https://bugs.launchpad.net/nova/+bug/1672792
[2] https://pythonhosted.org/six/#module-six.moves.urllib.parse
[3] https://docs.python.org/3/library/urllib.parse.html#urllib.parse.unquote

Correction to the above: six.text_type always represents unicode, so we would want to return the native text type depending on python version, not six.text_type.

That said, after playing around with a unit test, I'm not seeing that six.moves.urllib.parse.unquote returns type unicode -- it's returning type str. So again it's not yet clear where the unicode type is coming from.

Okay, I've done more sleuthing and found where the unicode type is coming from. It's coming from the instance.uuid -- instance.uuid is of type unicode in python2. So when the imagebackend.py code for rbd does this: "self.rbd_name = '%s_%s' % (instance.uuid, disk_name)" [1], because the instance.uuid is of type unicode, self.rbd_name is also unicode when it's assigned.

I tried some testing with a functional test and found that when an instance is queried from the database, its string fields are returned as type unicode. Here is a print statement I added in nova/objects/instance.py:

 358 def _from_db_object(context, instance, db_inst, expected_attrs=None):
 359 """Method to help with migration to objects.
 360
 361 Converts a database entity to a formal object.
 362 """
 363 instance._context = context
 364 if expected_attrs is None:
 365 expected_attrs = []
 366 # Most of the field names match right now, so be quick
 367 for field in instance.fields:
 368 if field in INSTANCE_OPTIONAL_ATTRS:
 369 continue
 370 elif field == 'deleted':
 371 instance.deleted = db_inst['deleted'] == db_inst['id']
 372 elif field == 'cleaned':
 373 instance.cleaned = db_inst['cleaned'] == 1
 374 else:
 375 instance[field] = db_inst[field]
 376 if field in ('uuid', 'project_id'):
 377 print('%s was of type: %s' % (field, type(db_inst[field])))

And the lines I added in a functional test:

 47 insts = objects.InstanceList.get_all(self.context.elevated())
 48 for i in insts:
 49 print(type(i.uuid))

and the output:

    project_id was of type: <type 'unicode'>
    uuid was of type: <type 'unicode'>
    project_id was of type: <type 'unicode'>
    uuid was of type: <type 'unicode'>
    project_id was of type: <type 'unicode'>
    uuid was of type: <type 'unicode'>
    project_id was of type: <type 'unicode'>
    uuid was of type: <type 'unicode'>
    <type 'unicode'>
    <type 'unicode'>
    <type 'unicode'>
    <type 'unicode'>

[1] https://github.com/openstack/nova/blob/372f624/nova/virt/libvirt/imagebackend.py#L807

I wondered why this isn't a problem in the legacy-tempest-dsvm-full-devstack-plugin-ceph job in the gate and I also couldn't reproduce the issue with devstack installed with the devstack-plugin-ceph today (which installs Ceph Luminous 12.2.4).

I realized the problem is that the bug reporter has Ceph Infernalis (or older) installed and the python bindings from back then [1] do not handle the unicode string type, whereas the more recent bindings for Ceph >= Jewel [2] work fine.

I just checked our gate job for Ocata [3], which installs Ceph Hammer, and python-rbd 10.2.9 gets installed along with it, and that also works fine without the "TypeError: name must be a string".

I agree with Dmitriy that this sounds like more of an issue with the python-rbd/python-rados versions that openstack-ansible is installing for Ceph. I think it would be a good idea to open a bug with them and try to get that part resolved.

Ceph Hammer is pretty old at this point but it is running in the Ocata gate and working fine with python-rbd/python-rados 10.2.9. Are you able to install that version of the bindings? What version of Ceph are you running?

In Nova, we could add more str() calls to be more resilient about old python binding versions in case someone is running something really old.

I'm going to update the importance of this bug to Low.

[1] https://github.com/ceph/ceph/blob/infernalis/src/pybind/rbd.py#L379-L416
[2] https://github.com/ceph/ceph/blob/luminous/src/pybind/rbd/rbd.pyx#L1343-L1380
[3] http://logs.openstack.org/77/588077/1/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/a2a2d25/logs/devstacklog.txt.gz#_2018-08-01_20_55_45_676

We are using ceph luminous as a back-end. Moreover, I've catched similar problem on one computing node during unshelving procedure - nova fails to unshelve an instance with "TypeError: name must be a string" on the image restoration step. I'm currently on some kind of vacation, so I'll be able to provide log output only within several days.
I will try to dig into openstack-ansible-os_nova to get why so old rbd is get installed, as luminous ceph version had been specified in user_variables.yml (and ceph client is installed from luminous outside of virtualenv).

It seems, that the problem lies not in the openstack-ansible, but rather in ceph repos.

OSA installs python-ceph package from ceph repo here:
https://github.com/openstack/openstack-ansible-ceph_client/blob/e67c08ca99e2f951d517af8e0735a90177f2e3ab/vars/ubuntu.yml#L43

python-ceph has python-rbd in its dependencies, and python-rbd provides rbd library, which is symlinked into virtualenv. However, deb package python-rbd of version 12.2.7 has python rbd library of version 2.0:

root@nova01:~# dpkg -l | grep python-rbd
ii python-rbd 12.2.7-1xenial amd64 Python 2 libraries for the Ceph librbd library
root@nova01:~# dpkg-query -L python-rbd
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/python-rbd
/usr/share/doc/python-rbd/copyright
/usr/share/doc/python-rbd/changelog.Debian.gz
/usr/lib
/usr/lib/python2.7
/usr/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages/rbd.x86_64-linux-gnu.so
/usr/lib/python2.7/dist-packages/rbd-2.0.0.egg-info
/usr/lib/python2.7/dist-packages/rbd-2.0.0.egg-info/dependency_links.txt
/usr/lib/python2.7/dist-packages/rbd-2.0.0.egg-info/top_level.txt
/usr/lib/python2.7/dist-packages/rbd-2.0.0.egg-info/PKG-INFO
root@nova01:~# apt show python-rbd
Package: python-rbd
Version: 12.2.7-1xenial
Priority: optional
Section: python
Source: ceph
Maintainer: Ceph Maintainers <email address hidden>
Installed-Size: 461 kB
Depends: librbd1 (>= 12.2.7-1xenial), python (<< 2.8), python (>= 2.7~), libc6 (>= 2.4), libpython2.7 (>= 2.7)
Breaks: python-ceph (<< 0.92-1223)
Replaces: python-ceph (<< 0.92-1223)
Homepage: http://ceph.com/
Download-Size: 127 kB
APT-Manual-Installed: no
APT-Sources: http://download.ceph.com/debian-luminous xenial/main amd64 Packages
Description: Python 2 libraries for the Ceph librbd library
 Ceph is a massively scalable, open-source, distributed
 storage system that runs on commodity hardware and delivers object,
 block and file system storage.
 .
 This package contains Python 2 libraries for interacting with Ceph's
 RBD block device library.

N: There are 3 additional records. Please use the '-a' switch to see them.
root@uacloud-nova01:~#

So it seems, that this bug has almost nothing to do with OSA. And it seems, that everyone, who will install rbd library not from pip (or sources) will face with this bug...

Well, the tempest ceph job in the gate installs packages with apt-get (if ubuntu) [1][2], so I don't think there's an issue of pip or sources, etc. The packages are coming from the Ubuntu Cloud Archive repo.

And the version you show of your python-rbd package is 12.2.7, which according to the 12.2.7 source [3], uses the cstr() function and should be handling unicode.

I don't know what the "version 2.0" of the library means but it seems unrelated to the actual version of python-rbd library being used. I have similar in my environment:

$ dpkg -l | grep python-rbd
ii python-rbd 12.2.4-0ubuntu1~cloud1 amd64 Python libraries for the Ceph librbd library

$ dpkg-query -L python-rbd
/.
/usr
/usr/lib
/usr/lib/python2.7
/usr/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages/rbd-2.0.0.egg-info
/usr/lib/python2.7/dist-packages/rbd-2.0.0.egg-info/PKG-INFO
/usr/lib/python2.7/dist-packages/rbd-2.0.0.egg-info/dependency_links.txt
/usr/lib/python2.7/dist-packages/rbd-2.0.0.egg-info/top_level.txt
/usr/lib/python2.7/dist-packages/rbd.x86_64-linux-gnu.so
/usr/share
/usr/share/doc
/usr/share/doc/python-rbd
/usr/share/doc/python-rbd/copyright
/usr/share/doc/python-rbd/changelog.Debian.gz

$ apt show python-rbd
Package: python-rbd
Version: 12.2.4-0ubuntu1~cloud1
Priority: optional
Section: python
Source: ceph
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Ceph Maintainers <email address hidden>
Installed-Size: 501 kB
Depends: librbd1 (= 12.2.4-0ubuntu1~cloud1), python (<< 2.8), python (>= 2.7~), libc6 (>= 2.4), libpython2.7 (>= 2.7)
Breaks: python-ceph (<< 0.94.1-1~)
Replaces: python-ceph (<< 0.94.1-1~)
Download-Size: 132 kB
APT-Manual-Installed: no
APT-Sources: http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/queens/main amd64 Packages
Description: Python libraries for the Ceph librbd library
 Ceph is a massively scalable, open-source, distributed
 storage system that runs on commodity hardware and delivers object,
 block and file system storage.
 .
 This package contains Python libraries for interacting with Ceph's
 RBD block device library.

N: There are 3 additional records. Please use the '-a' switch to see them.

And instances do *not* fail to create for unicode problems.

It seems like there is something wrong with the build of the packages that are available in the Ceph repo. The packages available in the Ubuntu Cloud Archive seem to be working well.

Maybe as a test, could you try installing python-rbd 12.2.7 from the Ubuntu Cloud Archive and see if it works without the TypeError? That would pinpoint if it's a problem with the packages from the Ceph repo and we could contact the maintainers to try to get it fixed. Because so far, it sounds like the package from the Ceph repo is including a much older version of rbd.py not matching 12.2.7.

[1] https://github.com/openstack/devstack-plugin-ceph/blob/21de8038dd5ea9af11c0c7b59daebebc2e723c2c/devstack/lib/ceph#L909
[2] https://github.com/openstack-dev/devstack/blob/ce0759d2d0030f5e578a6ebd648eed827135e584/functions-common#L1266-L1285
[3] https://github.com/ceph/ceph/blob/v12.2.7/src/pybind/rbd/rbd.pyx#L1343-L1381

Yep, you're right. I also can not reproduce the problem neither on the fresh installation, nor after upgrading packages on the existing ones. So probably this problem is already in the past, and new users shouldn't face with it (unless, they're using very old ceph, which is unlikely).

Thanks for confirming that. Maybe there was a bad package push to the repo and you got unlucky during that window when you installed it, or something else happened in your environment.

Yeah, I'm not opposed to the idea of being more resilient on the nova side but it seems highly unlikely that anyone is running python-rbd 10.0.1 or older with Rocky (unicode handling was added in version 10.0.2). As seen in the gate, even older Ceph Hammer works fine with python-rbd/python-rados 10.2.9. But these days many are running Luminous. And we don't want to carry workarounds for very old ceph bindings forever.