tripleo

tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 jobs are failing overcloud deployment with 'KeyError: u'type''

Bug #1824579 reported by Ronelle Landy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
yatin
tripleo stein-rc1

Bug Description

featureset039 jobs are failing in check and promotion. We are seeing errors in overcloud-prep-images (introspection errors - which may be cloud related) and overcloud deploy errors:

2019-04-12 17:31:32.066 8 ERROR heat.engine.check_resource [req-a799f5c3-5a74-4b9e-9c2e-e36f5cc7dea4 - admin - default default] Unexpected exception in resource check.: KeyError: u'type'

 [overcloud.ComputeServiceChain.ServiceServerMetadataHook]: CREATE_FAILED resources.ServiceServerMetadataHook: u'type'

2019-04-12 16:31:11 | 2019-04-12 16:30:49Z [ServiceServerMetadataHook.IncomingMetadataSettings]: CREATE_COMPLException occured while running the command

Full logs linked below:
(stein promotion)
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-stein/415c446/logs/undercloud/var/log/containers/heat/heat-engine.log.txt.gz#_2019-04-12_17_31_32_066

(master check)tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 jobs are failing overcloud deployme
https://logs.rdoproject.org/39/651239/10/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/ae8f974/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

The only passing fs039 jobs we see are on rocky - all other branches show failures

Ronelle Landy (rlandy)
tags: added: alert ci promotion-blocker
Changed in tripleo:
importance: Undecided → Critical
status: New → Triaged
Revision history for this message
wes hayutin (weshayutin) wrote :

just a guess, but maybe something that merged from https://review.openstack.org/#/q/topic:containers-common+(status:open+OR+status:merged) ?

Revision history for this message
Ronelle Landy (rlandy) wrote :

https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039&branch=master&result=FAILURE shows failing jobs since 04/11

Revision history for this message
Ronelle Landy (rlandy) wrote :

possible culprits:

<ykarel> <weshay> rlandy|ruck or https://review.openstack.org/#/c/639119/
...
<ykarel> that patch should have missed some tls config

Changed in tripleo:
milestone: none → stein-rc1
Revision history for this message
Ronelle Landy (rlandy) wrote :

see results from revert: https://review.openstack.org/#/c/652140/

Revision history for this message
Dan Prince (dan-prince) wrote :

I think I see the issue here. Its because I swapped out a hiera data lookup in t-h-t for OVNSouthboundServerPort in that commit. Let me see if I can post a fix here shortly.

Changed in tripleo:
assignee: nobody → Dan Prince (dan-prince)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/652437

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/652445

Revision history for this message
Dan Prince (dan-prince) wrote :

2 options there ^^, maybe we can use one of them to fix this.

Revision history for this message
yatin (yatinkarel) wrote :

<< I think I see the issue here. Its because I swapped out a hiera data lookup in t-h-t for OVNSouthboundServerPort in that commit. Let me see if I can post a fix here shortly.

Not sure if ^^ is the actual issue considering only tls(fs039) job is affected. I mean if this(String/Integer) is the cause why only tls job affected?
The issue i think is the change https://review.openstack.org/#/c/639119/16/deployment/ovn/ovn-metadata-container-puppet.yaml@284, looking at code it seems it looks for .type at when metadata_settings is not null(currently {}) http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/extraconfig/nova_metadata/krb-service-principals.j2.yaml#n68, i can fire up a patch to confirm this meanwhile.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/652499

Changed in tripleo:
assignee: Dan Prince (dan-prince) → yatin (yatinkarel)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/652499
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=23352806b64f4b50a1050426844ad4ce54da6248
Submitter: Zuul
Branch: master

commit 23352806b64f4b50a1050426844ad4ce54da6248
Author: yatin <email address hidden>
Date: Mon Apr 15 08:02:27 2019 +0530

    Fix OVN yaql KeyError: type error

    https://review.openstack.org/#/c/639119/ changed
    the 'metadata_settings' RoleData for ovn-metadata
    to {} which caused a TypeError, this patch reverts
    the metadata_settings to original to fix the issue.

    Closes-Bug: #1824579
    Change-Id: I09ed9c99065c946760c5319086ff0dbccd29a85c

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Dan Prince (<email address hidden>) on branch: master
Review: https://review.openstack.org/652437

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Dan Prince (<email address hidden>) on branch: master
Review: https://review.openstack.org/652445

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.5.0

This issue was fixed in the openstack/tripleo-heat-templates 10.5.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.