newton: tripleo-quickstart jobs don't build packages from stable/newton

Bug #1674067 reported by Michele Baldessari
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Attila Darazs

Bug Description

In backporting https://review.openstack.org/#/c/447229/ to stable/newton I consistently see the following CI error:
http://logs.openstack.org/29/447229/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/c1f6ba4/logs/quickstart_install.txt.gz

TASK [build-test-packages : Run DLRN] ******************************************
task path: /home/jenkins/workspace/gate-tripleo-ci-centos-7-nonha-multinode-oooq/.quickstart/usr/local/share/ansible/roles/build-test-packages/tasks/dlrn-build.yml:53
Saturday 18 March 2017 22:01:04 +0000 (0:00:01.182) 0:04:16.893 ********
fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set +e; source /home/jenkins/dlrn-venv/bin/activate; while true; do\n dlrn --config-file projects.ini --head-only --package-name openstack-tripleo-heat-templates --local --info-repo rdoinfo --dev;\n if [ $? -eq 0 ]; then\n # SUCCESS\n break;\n elif [ $? -eq 1 ]; then\n # FAILED\n exit 1;\n elif [ $? -eq 2 ]; then\n # RETRY\n continue;\n fi;\n # Unexpected DLRN return code\n exit $?;\n done;", "delta": "0:02:02.897140", "end": "2017-03-18 22:03:08.844269", "failed": true, "rc": 1, "start": "2017-03-18 22:01:05.947129", "stderr": "2017-03-18 22:01:06,817 INFO:dlrn-repositories:Getting https://github.com/rdo-packages/tripleo-heat-templates-distgit.git to ./data/openstack-tripleo-heat-templates_distro (rpm-newton)\n2017-03-18 22:01:07,895 INFO:dlrn-repositories:Falling back to rpm-master\n2017-03-18 22:01:08,206 INFO:dlrn-repositories:Getting git://git.openstack.org/openstack/tripleo-heat-templates to ./data/openstack-tripleo-heat-templates (stable/newton)\n2017-03-18 22:01:08,504 INFO:dlrn-build:Processing openstack-tripleo-heat-templates 50f0b974c8fd764c8a0cac5124d28e8382a73658\n2017-03-18 22:03:07,138 ERROR:dlrn-build:cmd failed. See logs at: /home/jenkins/DLRN/data/repos/50/f0/50f0b974c8fd764c8a0cac5124d28e8382a73658_dev/\n2017-03-18 22:03:07,139 INFO:dlrn:Received exception Error in build_rpm_wrapper for openstack-tripleo-heat-templates: \n\n RAN: /usr/bin/env /home/jenkins/DLRN/scripts/build_rpm.sh centos openstack-tripleo-heat-templates /home/jenkins/DLRN/data/repos/50/f0/50f0b974c8fd764c8a0cac5124d28e8382a73658_dev /home/jenkins/DLRN/data https://trunk.rdoproject.org/centos7-newton /home/jenkins/DLRN/data/openstack-tripleo-heat-templates_distro\n\n STDOUT:\n\n\n STDERR:\n\n2017-03-18 22:03:07,144 ERROR:dlrn:Received exception Error in build_rpm_wrapper for openstack-tripleo-heat-templates: \n\n RAN: /usr/bin/env /home/jenkins/DLRN/scripts/build_rpm.sh centos openstack-tripleo-heat-templates /home/jenkins/DLRN/data/repos/50/f0/50f0b974c8fd764c8a0cac5124d28e8382a73658_dev /home/jenkins/DLRN/data https://trunk.rdoproject.org/centos7-newton /home/jenkins/DLRN/data/openstack-tripleo-heat-templates_distro\n\n STDOUT:\n\n\n STDERR:\n\n2017-03-18 22:03:07,149 INFO:dlrn-notifications:Skipping notify email to ['<email address hidden>', '<email address hidden>']", "stdout": "", "stdout_lines": [], "warnings": []}

The error is:
http://logs.openstack.org/29/447229/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/c1f6ba4/logs/delorean_logs/50/f0/50f0b974c8fd764c8a0cac5124d28e8382a73658_dev/build.log.txt.gz

+ cp -ar scripts /builddir/build/BUILDROOT/openstack-tripleo-heat-templates-5.2.1-0.20170318220252.50f0b97.el7.centos.x86_64//usr/share/openstack-tripleo-heat-templates
cp: cannot stat 'scripts': No such file or directory
RPM build errors:
error: Bad exit status from /var/tmp/rpm-tmp.EcBM03 (%install)
    Bad exit status from /var/tmp/rpm-tmp.EcBM03 (%install)
Child return code was: 1
EXCEPTION: [Error()]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 88, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/util.py", line 547, in do
    raise exception.Error("Command failed. See logs for output.\n # %s" % (command,), child.returncode)
Error: Command failed. See logs for output.
 # bash --login -c /usr/bin/rpmbuild -bb --target x86_64 --nodeps /builddir/build/SPECS/openstack-tripleo-heat-templates.spec

Revision history for this message
Michele Baldessari (michele) wrote :

Reproducer:
mkdir -p /tmp/test && cd /tmp/test
git clone https://github.com/openstack-infra/tripleo-ci
cd tripleo-ci/
TRIPLEO_ROOT=/tmp/test ./scripts/tripleo.sh --delorean-setup
STABLE_RELEASE=newton TRIPLEO_ROOT=/tmp/test ./scripts/tripleo.sh --delorean-build openstack/tripleo-heat-templates

Revision history for this message
Michele Baldessari (michele) wrote :

No idea what is wrong because https://github.com/rdo-packages/tripleo-heat-templates-distgit/blob/newton-rdo/openstack-tripleo-heat-templates.spec does not have any mention of 'cp scripts', so we must be pulling some wrong branch

tags: added: ci
Revision history for this message
Michele Baldessari (michele) wrote :

I wonder if the issue is actually here: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/dlrn-build.yml#L2-L6 we set the branch for rdo to 'rpm-{stablebranchname}', but I don't see any rpm-newton branch here https://github.com/rdo-packages/tripleo-heat-templates-distgit/blob/newton-rdo/openstack-tripleo-heat-templates.spec only one called 'newton-rdo', so maybe that is the root of the problem?

tags: added: alert
Changed in tripleo:
milestone: ongoing → pike-1
Revision history for this message
Alan Pevec (apevec) wrote :

Michele, you're right, this was fixed in tripleo.sh by https://review.openstack.org/399578
and followups: https://review.openstack.org/424637 and https://review.openstack.org/441960

Attila Darazs (adarazs)
Changed in tripleo:
assignee: nobody → Attila Darazs (adarazs)
tags: added: quickstart
summary: - stable/newton delorean fails to build tripleo-heat-templates
+ stable/newton: delorean fails to build tripleo-heat-templates on oooq
+ jobs
summary: - stable/newton: delorean fails to build tripleo-heat-templates on oooq
- jobs
+ newton: puppet-tripleo fails to build in delorean, it requires puppet-
+ etcd
summary: - newton: puppet-tripleo fails to build in delorean, it requires puppet-
- etcd
+ newton: tripleo-quickstart jobs don't build packages from stable/newton
Revision history for this message
Ben Kero (ben-kero) wrote :

I have proposed a change to tripleo-quickstart-extras build-test-packages role similar to the tripleo.sh fix in this change: https://review.openstack.org/399578

In my local testing this results in a newton package of tripleo-quickstart-extras being built successfully.

You can see my change here: https://review.openstack.org/#/c/447747/

As an explanation, see comment 3. Michele is correct that the branch name of the package repository changed for the newton release to the format 'newton-rdo' from 'rpm-mitaka'. See the branches listed here for an example: https://github.com/rdo-packages/tripleo-heat-templates-distgit

The change adds a conditional to check for the newton release and sets the branch name to the appropriate value.

I don't have an explanation of why this isn't happening on ocata though, as that also is using the new 'ocata-rdo' branch name.

Revision history for this message
Attila Darazs (adarazs) wrote :

Sorry folks, there is supposed to be a Gerrit - Launchpad integration that seems to have failed this time.

Here's the correct fix, submitted just half an hour after I took ownership of the bug:

https://review.openstack.org/447530

I added the "Closes-Bug", so we should look into why this integration doesn't work.

@Ben: you can abandon your change, looks similar to mine but it doesn't handle the newer releases that come after newton (only master and mitaka is using the rpm-$VERSION format now, and every stable branch going forward is $VERSION-rdo)

I'll submit the change as soon as it passes gates. Gates were so backed up yesterday that I couldn't submit it before going to sleep (the last gate job finished 8 hours after I submitted the change).

Revision history for this message
Michele Baldessari (michele) wrote :

Since https://review.openstack.org/#/c/447530/ merged I will close this. Thanks everyone

Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-quickstart-extras 2.1.0

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-quickstart-extras 2.1.1

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.