tripleo

periodic centos-8-ovb featureset039-master fails overcloud ipa-client install No such file or directory

Bug #1875353 reported by Marios Andreou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Incomplete
High
Amol Kahat

Bug Description

At [1][2] the periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master fails during overcloud deploy running the root/setup-ipa-client.sh on the controllers:

          2020-04-26 19:07:29 | TASK [run enrollment script]
          2020-04-26 19:07:30 | fatal: [overcloud-controller-0]: FAILED! => changed=true
          2020-04-26 19:07:30 | cmd: /root/setup-ipa-client.sh >> /var/log/setup-ipa-client-ansible.log 2>&1
          2020-04-26 19:07:30 | fatal: [overcloud-controller-2]: FAILED! => changed=true

Looking more closely the actual error is in the ipaclient-install.log [3]

          2020-04-26T19:07:30Z DEBUG Backing up system configuration file '/
        etc/hostname'
          2020-04-26T19:07:30Z ERROR [Errno 2] No such file or directory: '/var/lib/ipa-client/sysrestore/d7bbe230ea6082f548e8f8faf1965356127b9ecf1466ca658cc72552dc408687-hostname'

This is likely some race/timing issue because it is not happening every run looking at zuul builds [4] at time of filing (attaching screenshot) there are green runs too so not consistent.

[1] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/a7ce16f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[2] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/e5a6722/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[3] https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/a7ce16f/logs/overcloud-controller-0/var/log/ipaclient-install.log.txt.gz
[4] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master

Tags: ci
Revision history for this message
Marios Andreou (marios-b) wrote :
Revision history for this message
Marios Andreou (marios-b) wrote :

16:51 < akahat|ruck> marios, hey.. you reported this issue: https://bugs.launchpad.net/tripleo/+bug/1875353
16:51 <@openstack> Launchpad bug 1875353 in tripleo "(inconsistent/race) periodic centos-8-ovb featureset039-master fails overcloud
                   ipa-client install No such file or directory: /var/lib/ipa-client/sysrestore" [High,Triaged]
16:51 < akahat|ruck> i think there is problem with the novajoin.
16:52 < akahat|ruck> it still uses python instead of python3. I think those small failures are leading to failure of ipa-install.
16:52 < akahat|ruck> marios, https://opendev.org/x/novajoin/src/branch/master/files/cloud-config-novajoin.yaml#L52
16:53 < akahat|ruck> I found one more job with the same failures.
16:53 < akahat|ruck> https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/4558612/logs/overcloud-controller-1/var/log/setup-ipa-client-ansible.log.txt.gz

Revision history for this message
Amol Kahat (amolkahat) wrote :

WIP: https://review.opendev.org/#/c/725465/

Changed in tripleo:
assignee: nobody → amolkahat (amolkahat)
Amol Kahat (amolkahat)
Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Grzegorz Grasza (xek) wrote :

I checked, that the directory should be provided by ipa-client-common package.

$ rpm -ql ipa-client-common | grep sysrestore
/var/lib/ipa-client/sysrestore

Marios Andreou (marios-b)
summary: - (inconsistent/race) periodic centos-8-ovb featureset039-master fails
+ (inconsistent) periodic centos-8-ovb featureset039-master fails
overcloud ipa-client install No such file or directory: /var/lib/ipa-
client/sysrestore
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Revision history for this message
Pooja Jadhav (poojajadhav22) wrote : Re: (inconsistent) periodic centos-8-ovb featureset039-master fails overcloud ipa-client install No such file or directory: /var/lib/ipa-client/sysrestore
Download full text (5.0 KiB)

This FS39 master job also alternatively failing with below error logs :

2020-06-03T02:53:29Z INFO Configured sudoers in /etc/authselect/user-nsswitch.conf
2020-06-03T02:53:29Z DEBUG File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 179, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 340, in run
    return cfgr.run()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 360, in run
    return self.execute()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 386, in execute
    for rval in self._executor():
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 655, in _configure
    next(executor)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "...

Read more...

Revision history for this message
Rafael Folco (rafaelfolco) wrote : Re: (inconsistent) periodic centos-8-ovb featureset039-master fails overcloud ipa-client install No such file or directory

from today's run:

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/5758322/logs/overcloud-controller-0/var/log/ipaclient-install.log.txt.gz

2020-06-05T11:10:06Z DEBUG The ipa-client-install command failed, exception: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/ipa-client/sysrestore/...'

consistently failing on ipa-client install...

summary: (inconsistent) periodic centos-8-ovb featureset039-master fails
- overcloud ipa-client install No such file or directory: /var/lib/ipa-
- client/sysrestore
+ overcloud ipa-client install No such file or directory
summary: - (inconsistent) periodic centos-8-ovb featureset039-master fails
- overcloud ipa-client install No such file or directory
+ periodic centos-8-ovb featureset039-master fails overcloud ipa-client
+ install No such file or directory
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: victoria-1 → victoria-3
Marios Andreou (marios-b)
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-3 → wallaby-rc1
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Revision history for this message
Marios Andreou (marios-b) wrote :

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

Changed in tripleo:
milestone: xena-1 → none
status: In Progress → Incomplete
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.