OpenStack Compute (nova)

inject_password fails with python3

Series train
Bug #1882421

Bug #1882421 reported by Dr. Jens Harbott on 2020-06-07

This bug affects 2 people

	Status	Importance	Assigned to
OpenStack Compute (nova)	Fix Released	Medium	Lee Yarwood
Train	In Progress	Undecided	Unassigned
Ussuri	Fix Released	Undecided	Unassigned
Victoria	Fix Released	Undecided	Unassigned
Wallaby	Fix Released	Undecided	Unassigned

Bug Description

Originally reported in #openstack-nova:

14:44 < lvdombrkr> hello guys, trying to inject admin_password (inject_password=true ) into image but when creating instance get this error in nova-compute.log
14:45 < lvdombrkr> 2020-06-06 14:53:50.188 6 WARNING nova.virt.disk.api [req-94f485ca-944c-40e9-bf14-c8b8dbe09a7b 052d02306e6746a4a3e7e5449de49f8c
413a4cadf9734fca9ec3e5e6192a446f - default default] Ignoring error injecting admin_password into image (a bytes-like object is required, not 'str')
14:45 < lvdombrkr> Train + Centos8

Can reproduce on master on devstack by installing python3-guestfs and setting

[libvirt]
inject_partition = -1
inject_password = true

in nova-cpu.conf. Backtrace after adding a hard "raise" into inject_data_into_fs():

Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.virt.libvirt.driver [None req-47214a25-b56a-4135-83bb-7c5ff4c86ca6 demo demo] [instance: 5604d60c-61c9-49b5-8786-ff5144817863] Error injecting data into image 4b3e63a6-b3c4-4de5-b515-cc286e7d5c48 (a bytes-like object is required, not 'str')
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [None req-47214a25-b56a-4135-83bb-7c5ff4c86ca6 demo demo] [instance: 5604d60c-61c9-49b5-8786-ff5144817863] Instance failed to spawn: TypeError: a bytes-like object is required, not 'str'
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] Traceback (most recent call last):
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/compute/manager.py", line 2614, in _build_resources
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] yield resources
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/compute/manager.py", line 2374, in _build_and_run_instance
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] self.driver.spawn(context, instance, image_meta,
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3604, in spawn
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] created_instance_dir, created_disks = self._create_image(
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3991, in _create_image
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] created_disks = self._create_and_inject_local_root(
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4119, in _create_and_inject_local_root
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] self._inject_data(backend, instance, injection_info)
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3894, in _inject_data
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] LOG.error('Error injecting data into image '
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] self.force_reraise()
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] six.reraise(self.type_, self.value, self.tb)
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] raise value
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3887, in _inject_data
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] disk_api.inject_data(disk.get_model(self._conn),
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/disk/api.py", line 368, in inject_data
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] return inject_data_into_fs(fs, key, net, metadata, admin_password,
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/disk/api.py", line 461, in inject_data_into_fs
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] inject_func(inject_val, fs)
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/disk/api.py", line 597, in _inject_admin_password_into_fs
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] new_shadow_data = _set_passwd(admin_user, admin_passwd,
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/disk/api.py", line 644, in _set_passwd
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] p_file = passwd_data.split("\n")
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] TypeError: a bytes-like object is required, not 'str'
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863]

Tags:

Dr. Jens Harbott (j-harbott) on 2020-06-07

Changed in nova:
assignee:	nobody → Dr. Jens Harbott (j-harbott)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-07: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/733982

Sylvain Bauza (sylvain-bauza) on 2020-06-08

Changed in nova:
importance:	Undecided → Low
tags:	added: libvirt

Balazs Gibizer (balazs-gibizer) on 2020-06-08

Changed in nova:
importance:	Low → Medium
tags:	added: compute removed: libvirt

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-13: Change abandoned on nova (master)

Change abandoned by Jens Harbott (frickler) (<email address hidden>) on branch: master
Review: https://review.opendev.org/733982
Reason: This is wrong, the issue happens only with guestfs, localfs seems to works with str properly. This exceeds my current capacity, so I'm abandoning to let others take over.

Dr. Jens Harbott (j-harbott) on 2020-06-13

Changed in nova:
assignee:	Dr. Jens Harbott (j-harbott) → nobody
status:	In Progress → Confirmed

Revision history for this message

Nathaniel Sherry (nsherry4) wrote on 2020-09-30:

Patch for reading files as str instead of bytes in nova/virt/disk/vfs/guestfs.py Edit (617 bytes, text/plain)

The issue seems to be that guestfs under Python 3 returns bytes rather than str for file reads while still accepting str for replacing and appending. This patch modifies the guestfs backend's read_file method to return str data for Python 3 while preserving existing behaviour for Python 2. I suppose as long as all the files being read are text, this should be fine.

Revision history for this message

Martin Kopec (mkopec) wrote on 2021-02-11:

We disabled file injection tests on tempest-all job due to this bug:
https://review.opendev.org/c/openstack/tempest/+/774701

Revision history for this message

Lee Yarwood (lyarwood) wrote on 2021-04-23:

https://github.com/libguestfs/libguestfs/commit/0ee02e0117527b86a31b2a88a14994ce7f15571f caused this, looking at a fix now.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-23: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/787712

Changed in nova:
status:	Confirmed → In Progress

Lee Yarwood (lyarwood) on 2021-04-23

Changed in nova:
assignee:	nobody → Lee Yarwood (lyarwood)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-24: Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/787712
Committed: https://opendev.org/openstack/nova/commit/606d588e3eca1d88ad26b4c2cfa3f2e1d5ed553e
Submitter: "Zuul (22348)"
Branch: master

commit 606d588e3eca1d88ad26b4c2cfa3f2e1d5ed553e
Author: Lee Yarwood <email address hidden>
Date: Fri Apr 23 12:14:42 2021 +0100

guestfs: With libguestfs >= v1.41.1 decode returned bytes to string

libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
of read_file from string to bytes.

https://github.com/libguestfs/libguestfs/commit/0ee02e0117527b86a31b2a88a14994ce7f15571f

    As we don't check the version of libguestfs installed this change
    handles both the original behaviour where a string is returned and the
    newer behaviour by decoding any returned bytes to a string.

Closes-Bug: #1882421
Change-Id: I1c12b2903c1e5bf3a88394493456ad33233f3cd8

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-24: Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/787839

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-25: Fix proposed to nova (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/787901

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-25: Fix proposed to nova (stable/ussuri)

#10

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/787902

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-25: Fix proposed to nova (stable/train)

#11

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/787903

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-26: Fix merged to nova (stable/wallaby)

#12

Reviewed: https://review.opendev.org/c/openstack/nova/+/787839
Committed: https://opendev.org/openstack/nova/commit/e98935f705a931e3ae666820387a419be1622b64
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit e98935f705a931e3ae666820387a419be1622b64
Author: Lee Yarwood <email address hidden>
Date: Fri Apr 23 12:14:42 2021 +0100

guestfs: With libguestfs >= v1.41.1 decode returned bytes to string

libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
of read_file from string to bytes.

https://github.com/libguestfs/libguestfs/commit/0ee02e0117527b86a31b2a88a14994ce7f15571f

    Closes-Bug: #1882421
    Change-Id: I1c12b2903c1e5bf3a88394493456ad33233f3cd8
    (cherry picked from commit 606d588e3eca1d88ad26b4c2cfa3f2e1d5ed553e)

tags:

added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-04-30: Fix merged to nova (stable/victoria)

#13

Reviewed: https://review.opendev.org/c/openstack/nova/+/787901
Committed: https://opendev.org/openstack/nova/commit/210abc09b857880aa7665fb540f46cb53662a600
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 210abc09b857880aa7665fb540f46cb53662a600
Author: Lee Yarwood <email address hidden>
Date: Fri Apr 23 12:14:42 2021 +0100

guestfs: With libguestfs >= v1.41.1 decode returned bytes to string

libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
of read_file from string to bytes.

https://github.com/libguestfs/libguestfs/commit/0ee02e0117527b86a31b2a88a14994ce7f15571f

    Closes-Bug: #1882421
    Change-Id: I1c12b2903c1e5bf3a88394493456ad33233f3cd8
    (cherry picked from commit 606d588e3eca1d88ad26b4c2cfa3f2e1d5ed553e)
    (cherry picked from commit e98935f705a931e3ae666820387a419be1622b64)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-14: Fix included in openstack/nova 23.0.1

#14

This issue was fixed in the openstack/nova 23.0.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-05-14: Fix included in openstack/nova 22.2.1

#15

This issue was fixed in the openstack/nova 22.2.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-07-14: Fix merged to nova (stable/ussuri)

#16

Reviewed: https://review.opendev.org/c/openstack/nova/+/787902
Committed: https://opendev.org/openstack/nova/commit/637402eec7f3f7f552bfc30d16c5960e3a651395
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 637402eec7f3f7f552bfc30d16c5960e3a651395
Author: Lee Yarwood <email address hidden>
Date: Fri Apr 23 12:14:42 2021 +0100

guestfs: With libguestfs >= v1.41.1 decode returned bytes to string

libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
of read_file from string to bytes.

https://github.com/libguestfs/libguestfs/commit/0ee02e0117527b86a31b2a88a14994ce7f15571f

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-17: Fix included in openstack/nova 24.0.0.0rc1

#17

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-07: Fix included in openstack/nova 21.2.3

#18

This issue was fixed in the openstack/nova 21.2.3 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-09-01: Change abandoned on nova (stable/train)

#19

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/787903
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.