TripleO-Quickstart hanging on virt-customize call during execution of modify-image role

Attempting to run quickstart against master rather than master-tripleo-ci. Maybe the latter's image was somehow corrupt.

Hash for the overlcloud image in question: 835667355e69ca32390c60f8fde2d676

This only seems to be affecting deployments against the master-tripleo-ci release. Run against master worked fine. Earlier releases are probably not affected.

This was resolved by manually yum updating the image in CI so that it does not need to update from 7.3->7.4.

I tested master-tripleo-ci release in quickstart this morning and it works fine.

Thanks for the speedy resolution, @trown!

We've got this in tripleo-quickstart-extras-gate-master-tripleo-ci-delorean-full-minimal_pacemaker job and I can notice a lot of [Errno 5] [Errno 12] Cannot allocate memory. Moreover, it ends with this error when running the `yum update`:

--> Finished Dependency Resolution
Error: Package: 1:grub2-efi-modules-2.02-0.44.el7.centos.x86_64 (@base)
           Requires: grub2-tools = 1:2.02-0.44.el7.centos
           Removing: 1:grub2-tools-2.02-0.44.el7.centos.x86_64 (installed)
               grub2-tools = 1:2.02-0.44.el7.centos
           Obsoleted By: 1:grub2-tools-minimal-2.02-0.64.el7.centos.x86_64 (quickstart-centos-base)
               Not found
           Installing: 1:grub2-tools-2.02-0.64.el7.centos.x86_64 (quickstart-centos-base)
               grub2-tools = 1:2.02-0.64.el7.centos
 You could try using --skip-broken to work around the problem
** Found 293 pre-existing rpmdb problem(s), 'yum check' output follows:

After a 240 min timeouts, here is the log of the repo_setup.sh script:
https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-extras-gate-master-tripleo-ci-delorean-full-minimal_pacemaker-3376/172.19.2.24/home/stack/_home_stack_repo_setup.sh.log.gz

Still seeing this issue https://ci.centos.org/view/rdo/view/tripleo-gate/job/tripleo-quickstart-extras-gate-master-tripleo-ci-delorean-full-minimal_pacemaker/3379/console

here too: https://ci.centos.org/artifacts/rdo/jenkins-tqe-containers-gate-master-tripleo-ci-delorean-full-containers_minimal-1653/console.txt.gz

I uploaded new overcloud-full image, and tested a local run.

Not entirely sure what happened between yesterday and today, but this should be resolved.

I'm seeing this today, with the image:

94f385d5742f61448587f21efcc56ddc.tar

which is from

https://images.rdoproject.org/master/rdo_trunk/3a65b17da7b98c83dbfda432af88bb56d3501de9_dba04735/overcloud-full.tar

It's a Skylake node, running CentOS 7.5, kernel 3.10.0-862.3.3

Appears to be related to the recent problems with libguestfs and 7.5 kernels, which for me are also apparently causing the undercloud VM not to boot properly.

Hi,

I'd appreciate if someone can provide a 'virt-customize' (with its full command-line) only reproducer here?

Adam Huffman says he's running the kernel: kernel 3.10.0-862.3.3. And alludes to "recent problems with libguestfs and 7.5". I just talked to libguestfs upstream folks, and they said, "there were some problems with initial Meltdown / Spectre fixes; but if you're running the latest CentOS kernel then you should be good".

What is the missing piece here?

    * * *

That said, I'm unable to reproduce the problem (in a RHEL environment that was provided by Michal Pryc from Red Hat; and here's the downstream bug: https://bugzilla.redhat.com/show_bug.cgi?id=1655542). And the following are three attempts (after some trial-and-error) at reproducers. All of them succeed at `yum update`.

Method-A1
---------

Get a fresh copy of 'overcloud-full.qcow2', and now try to run (notice
the "--skip-broken"):

    $ time virt-customize -v -x --selinux-relabel \
        --run-command 'yum -y update --skip-broken -d 10 -v' \
        -a v4-overcloud-full.qcow2 \
        |& tee method-A1.txt

The `yum update` update succeeds. The full log is in attachment.

Method-A2
---------

Again, get a fresh copy of 'overcloud-full.qcow2', and now try to run
(here we run *without* the "--skip-broken"):

    $ time virt-customize -v -x --selinux-relabel \
        --run-command 'yum -y update -d 10 -v' \
        -a v4-overcloud-full.qcow2 \
        |& tee method-A2.txt

Here, too. `yum update` update succeeds. The full log is in
attachment.

Method-B
--------

Trying to make `yum` run manually from the same OverCloud image in the
environment:

(1) On the UnderCloud environment, navigate to the "backup-overcloud"
    directory, and copy the QCOW2 file, 'vmlinuz' and 'initrd' into a
    directory called "v3"

    $ mkdir /home/stack/v3
    $ cd /home/stack/backup-overcloud/
    $ cp overcloud-full.qcow2 ~/v3/v3-overcloud-full.qcow2
    $ cp overcloud-full.vmlinuz overcloud-full.initrd ~/v3/

(2) Reset the root password to "empty" on the 'v3-overcloud-full.qcow2'

    $ cd /home/stack/v3/
    $ virt-edit -a v3-overcloud-full.qcow2 /etc/passwd -e 's/^root:.*?:/root::/'

(2) Then, import the 'v3-overcloud-full.qcow2' image (with the given
    'kernel' and 'initrd' into libvirt:

    $ sudo virt-install --name v3-overcloud-full-with-kernel \
    --ram 2048 --disk path=./v3-overcloud-full.qcow2,format=qcow2 \
    --machine q35 --os-variant fedora27 --cpu host-passthrough \
    --nographics --network default \
    --boot kernel=`pwd`/overcloud-full.vmlinuz,initrd=`pwd`/overcloud-full.initrd,kernel_args="panic=1 console=ttyS0 root=/dev/vda selinux=0" \
    --import

(3) Then, SSH into the guest, and run:

    $ yum update -y -d 10 -v

The `yum update` succeeds.