tripleo

tripleo modify-image role failed with format: raw error

Bug #2009488 reported by Amol Kahat on 2023-03-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	High	Unassigned	tripleo antelope-1

Bug Description

Tripleo Train ovb jobs failed with "mount: /tmp/tmp.5QARi9wSgW: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error."

Actual Error
============
2023-03-06 11:57:22.085603 | primary | TASK [modify-image : Mount image] **********************************************
2023-03-06 11:57:22.085676 | primary | Monday 06 March 2023 11:57:22 +0000 (0:00:03.166) 0:10:15.892 **********
2023-03-06 11:57:23.486873 | primary | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set -ex\nif type tripleo-mount-image >/dev/null; then\n tripleo-mount-image -a /home/zuul/overcloud-full.raw -m /tmp/tmp.5QARi9wSgW\nelse\n # stable branches do not have tripleo-mount-image, and only use\n # partition images\n modprobe nbd\n if qemu-img info --output json /home/zuul/overcloud-full.raw |grep '\"format\": \"raw\"' ; then\n image_format='--format raw'\n elif qemu-img info --output json /home/zuul/overcloud-full.raw |grep '\"format\": \"qcow2\"' ; then\n image_format='--format qcow2'\n else\n image_format=''\n fi\n qemu-nbd $image_format --connect /dev/nbd0 /home/zuul/overcloud-full.raw\n mount /dev/nbd0 /tmp/tmp.5QARi9wSgW\nfi\n", "delta": "0:00:00.104008", "end": "2023-03-06 11:57:23.303535", "msg": "non-zero return code", "rc": 32, "start": "2023-03-06 11:57:23.199527", "stderr": "+ type tripleo-mount-image\n/bin/sh: line 1: type: tripleo-mount-image: not found\n+ modprobe nbd\n+ qemu-img info --output json /home/zuul/overcloud-full.raw\n+ grep '\"format\": \"raw\"'\n+ image_format='--format raw'\n+ qemu-nbd --format raw --connect /dev/nbd0 /home/zuul/overcloud-full.raw\n+ mount /dev/nbd0 /tmp/tmp.5QARi9wSgW\nmount: /tmp/tmp.5QARi9wSgW: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.", "stderr_lines": ["+ type tripleo-mount-image", "/bin/sh: line 1: type: tripleo-mount-image: not found", "+ modprobe nbd", "+ qemu-img info --output json /home/zuul/overcloud-full.raw", "+ grep '\"format\": \"raw\"'", "+ image_format='--format raw'", "+ qemu-nbd --format raw --connect /dev/nbd0 /home/zuul/overcloud-full.raw", "+ mount /dev/nbd0 /tmp/tmp.5QARi9wSgW", "mount: /tmp/tmp.5QARi9wSgW: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error."], "stdout": " \"format\": \"raw\",", "stdout_lines": [" \"format\": \"raw\","]}
2023-03-06 11:57:23.487010 | primary | ...ignoring
2023-03-06 11:57:23.546052 | primary |
2023-03-06 11:57:23.546074 | primary | TASK [modify-image : Debug image mount] ****************************************
2023-03-06 11:57:23.546154 | primary | Monday 06 March 2023 11:57:23 +0000 (0:00:01.460) 0:10:17.352 **********
2023-03-06 11:57:23.689164 | primary | fatal: [undercloud]: FAILED! => {
2023-03-06 11:57:23.689198 | primary | "msg": {
2023-03-06 11:57:23.689206 | primary | "stderr": "+ type tripleo-mount-image\n/bin/sh: line 1: type: tripleo-mount-image: not found\n+ modprobe nbd\n+ qemu-img info --output json /home/zuul/overcloud-full.raw\n+ grep '\"format\": \"raw\"'\n+ image_format='--format raw'\n+ qemu-nbd --format raw --connect /dev/nbd0 /home/zuul/overcloud-full.raw\n+ mount /dev/nbd0 /tmp/tmp.5QARi9wSgW\nmount: /tmp/tmp.5QARi9wSgW: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.",
2023-03-06 11:57:23.689213 | primary | "stdout": " \"format\": \"raw\","
2023-03-06 11:57:23.689218 | primary | }
2023-03-06 11:57:23.689223 | primary | }

Logs
====
- https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/1ec5088/job-output.txt
- https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/ab9bc61/job-output.txt
- https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/68c9299/job-output.txt

Tags:

Revision history for this message

Amol Kahat (amolkahat) wrote on 2023-03-06:

Observing some strange results here[1], fs-39-train job is failing on that error, but in later run it got passed.

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train&skip=0

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2023-03-10:

tripleo-mount-image doesn't exist in train, so this is happening in the old fallback path: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/modify-image/tasks/manual.yml#L49-L60

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2023-03-10:

This could be due to less mature support from the kernel nbd module in the centos-8 used on train. I'm not sure what to recommend other than switching back to kpartx/loop-device for this fallback path

Revision history for this message

Ronelle Landy (rlandy) wrote on 2023-03-20:

https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-internal-train&skip=0 seems not to hit in the downstream