tripleo

Stein periodic featureset020 job is timing out on overcloud deployment - 1.5 hour gap in a single task

Bug #1850934 reported by Ronelle Landy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Ronelle Landy
tripleo ussuri-1

Bug Description

Stein periodic featureset020 jobs have been timing out in overcloud deployment. The logs show that there are tasks where 1.5 hours are spent:

See:

2019-11-01 06:37:56 | TASK [create persistent directories] *******************************************
2019-11-01 06:37:56 | changed: [overcloud-novacompute-1] => (item={u'path': u'/var/log/containers/neutron', u'setype': u'svirt_sandbox_file_t'}) => {
2019-11-01 06:37:56 | "ansible_loop_var": "item",
2019-11-01 06:37:56 | "changed": true,
2019-11-01 06:37:56 | "gid": 0,
2019-11-01 06:37:56 | "group": "root",
2019-11-01 06:37:56 | "item": {
2019-11-01 06:37:56 | "path": "/var/log/containers/neutron",
2019-11-01 06:37:56 | "setype": "svirt_sandbox_file_t"
2019-11-01 06:37:56 | },
2019-11-01 06:37:56 | "mode": "0755",
2019-11-01 06:37:56 | "owner": "root",
2019-11-01 06:37:56 | "path": "/var/log/containers/neutron",
2019-11-01 06:37:56 | "secontext": "unconfined_u:object_r:container_file_t:s0",
2019-11-01 06:37:56 | "size": 6,
2019-11-01 06:37:56 | "state": "directory",
2019-11-01 06:37:56 | "uid": 0
2019-11-01 06:37:56 | }
2019-11-01 06:37:56 | changed: [overcloud-novacompute-0] => (item={u'path': u'/var/log/containers/neutron', u'setype': u'svirt_sandbox_file_t'}) => {
2019-11-01 06:37:56 | "ansible_loop_var": "item",
2019-11-01 06:37:56 | "changed": true,
2019-11-01 06:37:56 | "gid": 0,
2019-11-01 06:37:56 | "group": "root",
2019-11-01 06:37:56 | "item": {
2019-11-01 06:37:56 | "path": "/var/log/containers/neutron",
2019-11-01 06:37:56 | "setype": "svirt_sandbox_file_t"
2019-11-01 06:37:56 | },
2019-11-01 06:37:56 | "mode": "0755",
2019-11-01 06:37:56 | "owner": "root",
2019-11-01 08:07:53 | "path": "/var/log/containers/neutron",

Full log:

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/1bfe724/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

And ...

2019-10-30 06:41:40 | RUNNING HANDLER [chrony : Restart chronyd] *************************************
2019-10-30 06:41:40 | changed: [overcloud-novacompute-1] => {
2019-10-30 06:41:40 | "changed": true,
2019-10-30 06:41:40 | "name": "chronyd",
2019-10-30 06:41:40 | "state": "started",
2019-10-30 06:41:40 | "status": {

<snip>
2019-10-30 06:41:40 | }
2019-10-30 06:41:40 | changed: [overcloud-novacompute-0] => {
2019-10-30 06:41:40 | "changed": true,
2019-10-30 06:41:40 | "name": "chronyd",
2019-10-30 06:41:40 | "state": "started",

<snip>

2019-10-30 06:41:40 | "IgnoreOnSnapshot": "no",
2019-10-30 06:41:40 | "IgnoreSIGPIPE": "yes",
2019-10-30 06:41:40 | "InactiveEnterTimestamp": "Wed 2019-10-30 06:41:29 UTC",
2019-10-30 06:41:40 | "InactiveEnterTimestampMonotonic": "460780200",
2019-10-30 06:41:40 | "InactiveExitTimestamp": "Wed 2019-10-30 06:41:29 UTC",
2019-10-30 08:10:34 | "InactiveExitTimestampMonotonic": "460785678",

Full log:

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/9d2b9bb/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

The OVB deploy timeout is already set to 2 hours.

Ronelle Landy (rlandy)
tags: added: ci promotion-blocker
Changed in tripleo:
milestone: none → ussuri-1
importance: Undecided → Critical
status: New → Triaged
assignee: nobody → Ronelle Landy (rlandy)
Revision history for this message
Ronelle Landy (rlandy) wrote :

Could possibly be fixed by https://review.opendev.org/#/c/691919/3/roles/undercloud-deploy/templates/containers-prepare-parameter.yaml.j2

Waiting on testproject results

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/692557

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/692558

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/692559

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/692598

Revision history for this message
Ronelle Landy (rlandy) wrote :

Passed on rerun:

periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein testproject master check 21850,9 16352 2019-11-01T13:45:54 SUCCESS

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/692557
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=61c40a7800a4d1b469832b7dfabf00b3072442b6
Submitter: Zuul
Branch: stable/stein

commit 61c40a7800a4d1b469832b7dfabf00b3072442b6
Author: Emilien Macchi <email address hidden>
Date: Mon Sep 23 19:14:41 2019 -0400

    Adapt ContainerImagePrepareDebug to the string pattern

    Like other *Debug parameters, make it so we first look for
    ContainerImagePrepareDebug to be set, otherwise we fallback to Debug;
    like we already do in all other OpenStack services.

    Related-Bug: #1850934
    Change-Id: I0f18b475c69a8ba71b06f517e87caf0d5c209fbb
    (cherry picked from commit 1d11972b10b8154d2e4a34c233a7c7c66048f0a9)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/692559
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fb494f110d6e02a729bf5b67c363845c0012981d
Submitter: Zuul
Branch: stable/stein

commit fb494f110d6e02a729bf5b67c363845c0012981d
Author: Alex Schultz <email address hidden>
Date: Fri Nov 1 08:51:41 2019 -0600

    Honor Debug for container image prepare

    We were never using Debug if ContainerImagePrepareDebug was not set.
    This change adds the Debug param and updates ContainerImagePrepareDebug
    to be '' by default. This makes it honor the Debug setting if
    ContainerImagePrepareDebug is not configured.

    Change-Id: I09b3b112d7654fed5270c0b1148b57b91b4a3215
    Related-Bug: #1850934
    (cherry picked from commit f70ba4bfa467d5622dc4d781a4d3b108064de6ae)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/692558
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f70ba4bfa467d5622dc4d781a4d3b108064de6ae
Submitter: Zuul
Branch: master

commit f70ba4bfa467d5622dc4d781a4d3b108064de6ae
Author: Alex Schultz <email address hidden>
Date: Fri Nov 1 08:51:41 2019 -0600

    Honor Debug for container image prepare

    We were never using Debug if ContainerImagePrepareDebug was not set.
    This change adds the Debug param and updates ContainerImagePrepareDebug
    to be '' by default. This makes it honor the Debug setting if
    ContainerImagePrepareDebug is not configured.

    Change-Id: I09b3b112d7654fed5270c0b1148b57b91b4a3215
    Related-Bug: #1850934

Revision history for this message
Ronelle Landy (rlandy) wrote :

Related bug: https://bugs.launchpad.net/tripleo/+bug/1849101

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/692598
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=eb6fd5354916f279ac980514ecedfd3ac9ca5618
Submitter: Zuul
Branch: stable/train

commit eb6fd5354916f279ac980514ecedfd3ac9ca5618
Author: Alex Schultz <email address hidden>
Date: Fri Nov 1 08:51:41 2019 -0600

    Honor Debug for container image prepare

    We were never using Debug if ContainerImagePrepareDebug was not set.
    This change adds the Debug param and updates ContainerImagePrepareDebug
    to be '' by default. This makes it honor the Debug setting if
    ContainerImagePrepareDebug is not configured.

    Change-Id: I09b3b112d7654fed5270c0b1148b57b91b4a3215
    Related-Bug: #1850934
    (cherry picked from commit f70ba4bfa467d5622dc4d781a4d3b108064de6ae)

tags: added: in-stable-train
Revision history for this message
Ronelle Landy (rlandy) wrote :

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein is doing better the last two days

Revision history for this message
Ronelle Landy (rlandy) wrote :

Going to close this out as stein is doing much better lately and debug has been added

Changed in tripleo:
status: Triaged → Fix Committed
Emilien Macchi (emilienm)
Changed in tripleo:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.