tripleo

ERROR! Unable to retrieve file contents, Could not find or access '/home/zuul/workspace/.quickstart/config/release/queens.yml' on the Ansible Controller. failing tripleo quickstart deployment on centos-8 (master, ussuri and train)

Bug #1880383 reported by Bhagyashri Shewale on 2020-05-24

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	High	wes hayutin	tripleo ussuri-rc3 "tripleo ussuri-rc3"

Bug Description

Standalone deployment jobs are failing on centos-8 (master, ussuri and train) with below error.

Error log:

2020-05-24 09:05:58.296700 | primary | ++(/home/zuul/src/opendev.org/openstack/tripleo-quickstart/ansible_ssh_env.sh:4): source(): SSH_CONFIG=/home/zuul/workspace/.quickstart/ssh.config.ansible
2020-05-24 09:05:58.296706 | primary | ++(/home/zuul/src/opendev.org/openstack/tripleo-quickstart/ansible_ssh_env.sh:6): source(): touch /home/zuul/workspace/.quickstart/ssh.config.ansible
2020-05-24 09:05:58.297525 | primary | ++(/home/zuul/src/opendev.org/openstack/tripleo-quickstart/ansible_ssh_env.sh:7): source(): export 'ANSIBLE_SSH_ARGS=-F /home/zuul/workspace/.quickstart/ssh.config.ansible'
2020-05-24 09:05:58.297549 | primary | ++(/home/zuul/src/opendev.org/openstack/tripleo-quickstart/ansible_ssh_env.sh:7): source(): ANSIBLE_SSH_ARGS='-F /home/zuul/workspace/.quickstart/ssh.config.ansible'
2020-05-24 09:05:58.297571 | primary | +(./quickstart.sh:539): main(): '[' 1 = 0 -a -z '' ']'
2020-05-24 09:05:58.297585 | primary | +(./quickstart.sh:548): main(): '[' '' = 1 ']'
2020-05-24 09:05:58.297592 | primary | +(./quickstart.sh:551): main(): VERBOSITY=vv
2020-05-24 09:05:58.297598 | primary | +(./quickstart.sh:554): main(): '[' '!' -f /home/zuul/workspace/.quickstart/playbooks/noop.yml ']'
2020-05-24 09:05:58.297605 | primary | +(./quickstart.sh:559): main(): ansible-playbook -vv /home/zuul/workspace/.quickstart/playbooks/noop.yml -e @/home/zuul/workspace/.quickstart/config/release/queens.yml -e @/home/zuul/workspace/.quickstart/config/nodes/1ctlr_1comp.yml -e @/home/zuul/workspace/.quickstart/config/general_config/minimal.yml -e @/home/zuul/workspace/.quickstart/config/environments/default_libvirt.yml -e local_working_dir=/home/zuul/workspace/.quickstart -e virthost=127.0.0.2 -t untagged,provision,environment,libvirt,undercloud-scripts,undercloud-inventory,overcloud-scripts,undercloud-setup,undercloud-install,undercloud-post-install,tripleoui-validate,notest,teardown-nodes
2020-05-24 09:05:59.054422 | primary | ansible-playbook 2.9.9
2020-05-24 09:05:59.054510 | primary | config file = /home/zuul/src/opendev.org/openstack/tripleo-quickstart/ansible.cfg
2020-05-24 09:05:59.054576 | primary | configured module search path = ['/usr/lib/python2.7/site-packages/tripleo-quickstart/library', '/home/zuul/workspace/.quickstart/usr/local/share/tripleo-quickstart/library', '/home/zuul/src/opendev.org/openstack/tripleo-quickstart/library']
2020-05-24 09:05:59.054624 | primary | ansible python module location = /home/zuul/workspace/.quickstart/lib64/python3.6/site-packages/ansible
2020-05-24 09:05:59.054634 | primary | executable location = /home/zuul/workspace/.quickstart/bin/ansible-playbook
2020-05-24 09:05:59.054639 | primary | python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
2020-05-24 09:05:59.054653 | primary | Using /home/zuul/src/opendev.org/openstack/tripleo-quickstart/ansible.cfg as config file
2020-05-24 09:05:59.085613 | primary | [WARNING]: Found both group and host with same name: undercloud
2020-05-24 09:05:59.085866 | primary | ERROR! Unable to retrieve file contents
2020-05-24 09:05:59.085883 | primary | Could not find or access '/home/zuul/workspace/.quickstart/config/release/queens.yml' on the Ansible Controller.
2020-05-24 09:05:59.085890 | primary | If you are using a module and expect the file to exist on the remote, see the remote_src option
2020-05-24 09:05:58.971402 | primary | ERROR
2020-05-24 09:05:58.971629 | primary | {
2020-05-24 09:05:58.971695 | primary | "delta": "0:01:34.327446",
2020-05-24 09:05:58.971741 | primary | "end": "2020-05-24 09:05:59.116242",
2020-05-24 09:05:58.971805 | primary | "msg": "non-zero return code",
2020-05-24 09:05:58.971900 | primary | "rc": 1,
2020-05-24 09:05:58.971982 | primary | "start": "2020-05-24 09:04:24.788796"
2020-05-24 09:05:58.972053 | primary | }
2020-05-24 09:05:59.083771 |
2020-05-24 09:05:59.084087 | PLAY RECAP
2020-05-24 09:05:59.084178 | primary | ok: 8 changed: 5 unreachable: 0 failed: 1 skipped: 11 rescued: 0 ignored: 0
2020-05-24 09:05:59.084223 |
2020-05-24 09:05:59.421390 | RUN END RESULT_NORMAL: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
2020-05-24 09:05:59.421814 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master]

Reference link:

[1]: https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-master/38ed2b3/job-output.txt

[2]: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master#

[3]: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released#

[4]: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-24hr#

Tags:

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-05-24:

I think the real error is just above.. pip failed on a dependency and caused a failure in the basic install.

2020-05-24 17:04:42.399115 | primary | ERROR: grafyaml 0.0.7 has requirement voluptuous<=0.10.5, but you'll have voluptuous 0.11.7 which is incompatible.
2020-05-24 17:04:42.399433 | primary | Installing collected packages: six, pyparsing, pyperclip, wcwidth, cmd2, MarkupSafe, jinja2, PyYAML, pycparser, cffi, cryptography, ansible, jmespath, netaddr, pbr, appdirs, os-service-types, munch, netifaces, requestsexceptions, decorator, dogpile.cache, urllib3, idna, certifi, chardet, requests, iso8601, stevedore, keystoneauth1, jsonpointer, jsonpatch, openstacksdk, os-client-config, setuptools, attrs, zipp, importlib-metadata, pyrsistent, jsonschema, warlock, futurist, clg, yamlordereddictloader, clif, msgpack, wrapt, debtcollector, pytz, Babel, oslo.i18n, oslo.utils, oslo.serialization, rfc3986, oslo.config, tripleo-quickstart-extras, tripleo-ha-utils, elasticsearch, voluptuous, text-unidecode, python-slugify, grafyaml, python-dateutil, PrettyTable, cliff, simplejson, osc-lib, python-novaclient, python-cinderclient, python-keystoneclient, python-openstackclient, docopt, pykwalify, browbeat, openstack-ansible-os-tempest, ansible-role-python-venv-build, ansible-config-template, ansible-role-collect-logs, openstack-tempest-skiplist, tripleo-operator-ansible, sqlalchemy, colorama, jenkinsapi, tripleo-ci, tripleo-upgrade

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-24: Fix proposed to tripleo-quickstart (master)

Fix proposed to branch: master
Review: https://review.opendev.org/730532

Changed in tripleo:
assignee:	nobody → wes hayutin (weshayutin)
status:	Triaged → In Progress

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-05-24:

https://review.opendev.org/730532

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-05-25:

OK... it's the toci-quickstart run quickstart script.
https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_quickstart.sh.j2#L47

ATM, we'll add queens to centos8 and put up a fix later.

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-05-25:

Download full text (3.7 KiB)

the noop.yml playbook which just gathers facts ( I think ) was failing to find the queens.yml release file.. I believe because of the distro.

WORKING EXAMPLE of a master job.. still calls queens.yml for noop.

2020-05-23 01:44:17.276230 | primary | +(./quickstart.sh:559): main(): ansible-playbook -vv /home/zuul/workspace/.quickstart/playbooks/noop.yml -e @/home/zuul/workspace/.quickstart/config/release/queens.yml -e @/home/zuul/workspace/.quickstart/config/nodes/1ctlr_1comp.yml -e @/home/zuul/workspace/.quickstart/config/general_config/minimal.yml -e @/home/zuul/workspace/.quickstart/config/environments/default_libvirt.yml -e local_working_dir=/home/zuul/workspace/.quickstart -e virthost=undercloud -t untagged,provision,environment,libvirt,undercloud-scripts,undercloud-inventory,overcloud-scripts,undercloud-setup,undercloud-install,undercloud-post-install,tripleoui-validate,notest,teardown-nodes
2020-05-23 01:44:18.179713 | primary | ansible-playbook 2.9.9
2020-05-23 01:44:18.179776 | primary | config file = /home/zuul/src/opendev.org/openstack/tripleo-quickstart/ansible.cfg
2020-05-23 01:44:18.179900 | primary | configured module search path = ['/usr/lib/python2.7/site-packages/tripleo-quickstart/library', '/home/zuul/workspace/.quickstart/usr/local/share/tripleo-quickstart/library', '/home/zuul/src/opendev.org/openstack/tripleo-quickstart/library', '/home/zuul/workspace/.quickstart/share/ansible/plugins/modules']
2020-05-23 01:44:18.180046 | primary | ansible python module location = /home/zuul/workspace/.quickstart/lib/python3.6/site-packages/ansible
2020-05-23 01:44:18.180067 | primary | executable location = /home/zuul/workspace/.quickstart/bin/ansible-playbook
2020-05-23 01:44:18.180076 | primary | python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
2020-05-23 01:44:18.180102 | primary | Using /home/zuul/src/opendev.org/openstack/tripleo-quickstart/ansible.cfg as config file
2020-05-23 01:44:18.216282 | primary | [WARNING]: Found both group and host with same name: undercloud
2020-05-23 01:44:19.466686 | primary |
2020-05-23 01:44:19.466757 | primary | PLAYBOOK: noop.yml *************************************************************
2020-05-23 01:44:19.466777 | primary | 1 plays in /home/zuul/workspace/.quickstart/playbooks/noop.yml
2020-05-23 01:44:19.469682 | primary |
2020-05-23 01:44:19.469710 | primary | PLAY [noop playbook] ***********************************************************
2020-05-23 01:44:19.481442 | primary |
2020-05-23 01:44:19.481470 | primary | TASK [Gathering Facts] *********************************************************
2020-05-23 01:44:19.481484 | primary | task path: /home/zuul/workspace/.quickstart/playbooks/noop.yml:4
2020-05-23 01:44:19.481636 | primary | Saturday 23 May 2020 01:44:19 +0000 (0:00:00.146) 0:00:00.146 **********
2020-05-23 01:44:20.689930 | primary | ok: [localhost]
2020-05-23 01:44:20.693379 | primary | META: ran handlers
2020-05-23 01:44:20.706179 | primary |
2020-05-23 01:44:20.706207 | primary | TASK [debug] *******************************************************************
2020-05-23 01:44:20.706221 | primary | task path: /home...

the noop.yml playbook which just gathers facts ( I think ) was failing to find the queens.yml release file.. I believe because of the distro.

WORKING EXAMPLE of a master job.. still calls queens.yml for noop.

2020-05-23 01:44:17.276230 | primary | +(./quickstart.sh:559): main(): ansible-playbook -vv /home/zuul/workspace/.quickstart/playbooks/noop.yml -e @/home/zuul/workspace/.quickstart/config/release/queens.yml -e @/home/zuul/workspace/.quickstart/config/nodes/1ctlr_1comp.yml -e @/home/zuul/workspace/.quickstart/config/general_config/minimal.yml -e @/home/zuul/workspace/.quickstart/config/environments/default_libvirt.yml -e local_working_dir=/home/zuul/workspace/.quickstart -e virthost=undercloud -t untagged,provision,environment,libvirt,undercloud-scripts,undercloud-inventory,overcloud-scripts,undercloud-setup,undercloud-install,undercloud-post-install,tripleoui-validate,notest,teardown-nodes
2020-05-23 01:44:18.179713 | primary | ansible-playbook 2.9.9
2020-05-23 01:44:18.179776 | primary |   config file = /home/zuul/src/opendev.org/openstack/tripleo-quickstart/ansible.cfg
2020-05-23 01:44:18.179900 | primary |   configured module search path = ['/usr/lib/python2.7/site-packages/tripleo-quickstart/library', '/home/zuul/workspace/.quickstart/usr/local/share/tripleo-quickstart/library', '/home/zuul/src/opendev.org/openstack/tripleo-quickstart/library', '/home/zuul/workspace/.quickstart/share/ansible/plugins/modules']
2020-05-23 01:44:18.180046 | primary |   ansible python module location = /home/zuul/workspace/.quickstart/lib/python3.6/site-packages/ansible
2020-05-23 01:44:18.180067 | primary |   executable location = /home/zuul/workspace/.quickstart/bin/ansible-playbook
2020-05-23 01:44:18.180076 | primary |   python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
2020-05-23 01:44:18.180102 | primary | Using /home/zuul/src/opendev.org/openstack/tripleo-quickstart/ansible.cfg as config file
2020-05-23 01:44:18.216282 | primary | [WARNING]: Found both group and host with same name: undercloud
2020-05-23 01:44:19.466686 | primary |
2020-05-23 01:44:19.466757 | primary | PLAYBOOK: noop.yml *************************************************************
2020-05-23 01:44:19.466777 | primary | 1 plays in /home/zuul/workspace/.quickstart/playbooks/noop.yml
2020-05-23 01:44:19.469682 | primary |
2020-05-23 01:44:19.469710 | primary | PLAY [noop playbook] ***********************************************************
2020-05-23 01:44:19.481442 | primary |
2020-05-23 01:44:19.481470 | primary | TASK [Gathering Facts] *********************************************************
2020-05-23 01:44:19.481484 | primary | task path: /home/zuul/workspace/.quickstart/playbooks/noop.yml:4
2020-05-23 01:44:19.481636 | primary | Saturday 23 May 2020  01:44:19 +0000 (0:00:00.146)       0:00:00.146 **********
2020-05-23 01:44:20.689930 | primary | ok: [localhost]
2020-05-23 01:44:20.693379 | primary | META: ran handlers
2020-05-23 01:44:20.706179 | primary |
2020-05-23 01:44:20.706207 | primary | TASK [debug] *******************************************************************
2020-05-23 01:44:20.706221 | primary | task path: /home/zuul/workspace/.quickstart/playbooks/noop.yml:7
2020-05-23 01:44:20.706354 | primary | Saturday 23 May 2020  01:44:20 +0000 (0:00:01.224)       0:00:01.371 **********
2020-05-23 01:44:20.720654 | primary | ok: [localhost] => {}
2020-05-23 01:44:20.720699 | primary |
2020-05-23 01:44:20.720728 | primary | MSG:
2020-05-23 01:44:20.720742 | primary |
2020-05-23 01:44:20.720754 | primary | noop
2020-05-23 01:44:20.721251 | primary | META: ran handlers
2020-05-23 01:44:20.721612 | primary | META: ran handlers
2020-05-23 01:44:20.723066 | primary |

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-05-25:

alternative patch for: https://review.opendev.org/#/c/730532/

https://review.opendev.org/730543

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-25: Change abandoned on tripleo-quickstart (master)

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.opendev.org/730532
Reason: didn't fix it

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-05-25:

hrm.. coming up empty on this one..

debugging w/ https://review.opendev.org/730553

Revision history for this message

chandan kumar (chkumar246) wrote on 2020-05-25:

https://review.opendev.org/#/c/730533/ - Revert "[install-deps] Use venv instead of virtualenv when possible" will fix the issue testing here: https://review.rdoproject.org/r/27739

Revision history for this message

chandan kumar (chkumar246) wrote on 2020-05-25:

#10

primary | ERROR: grafyaml 0.0.7 has requirement voluptuous<=0.10.5, but you'll have voluptuous 0.11.7 which is incompatible. needs to be fixed otherwise it will cause in future.

Revision history for this message

Bhagyashri Shewale (bhagyashri-shewale) wrote on 2020-05-25:

#11

On fix : https://review.opendev.org/#/c/730533/1 (Revert "[install-deps] Use venv instead of virtualenv when possible") standalone deployment jobs got *SUCCESS* https://review.rdoproject.org/r/#/c/27734/ ([DNM] Test bug/1880383)

Job Result:

Zuul CI 3:35 PM
noop SUCCESS in 0s
periodic-tripleo-ci-centos-8-standalone-master SUCCESS in 1h 09m 29s
periodic-tripleo-ci-centos-8-standalone-ussuri SUCCESS in 1h 12m 23s
periodic-tripleo-ci-centos-8-standalone-train SUCCESS in 59m 59s

Revision history for this message

Bhagyashri Shewale (bhagyashri-shewale) wrote on 2020-05-25:

#12

Hi All,

As fix got merge https://review.opendev.org/#/c/730533/1 i am closing this bug.

Changed in tripleo:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.