kolla

kolla-ansible fails to deploy heat if the heat_user_domain already exists in keystone

Bug #1839824 reported by Joseignacio-carretero on 2019-08-12

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	kolla	Fix Released	Medium	Radosław Piliszek	kolla 10.2.0 "ussuri"

Bug Description

I am deploying openstack Stein with kolla on Ubuntu 18.04. I have a keystone with data I want to keep from a previous Openstack installation. I already had a heat version installed and when I try to deploy Heat I get the following error:

+++ CURRENT_HEAT_DOMAIN_NAME='heat_user_domain
heat'
+++ [[ heat_user_domain != \\h\\e\\a\\t\\_\\u\\s\\e\\r\\_\\d\\o\\m\\a\\i\
\\
\\h\\e\\a\\t ]]
+++ openstack domain create heat_user_domain
Conflict occurred attempting to store domain - Duplicate Entry (HTTP 409) (Request-ID: req-c2e725bb-ebb8-40ef-9563-77977a6b959a)
", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Copying /var/lib/kolla/config_files/heat.conf to /etc/heat/heat.conf", "INFO:__main__:Setting permission for /etc/heat/heat.conf", "INFO:__main__:Writing out command to execute", "++ cat /run_command", "+ CMD=heat-api", "+ ARGS=", "+ [[ ! -n '' ]]", "+ . kolla_extend_start", "++ [[ ! -d /var/log/kolla/heat ]]", "++ mkdir -p /var/log/kolla/heat", "+++ stat -c %a /var/log/kolla/heat", "++ [[ 2755 != \\7\\5\\5 ]]", "++ chmod 755 /var/log/kolla/heat", "+++ whoami", "++ [[ heat == \\r\\o\\o\\t ]]", "++ . /usr/local/bin/kolla_heat_extend_start", "+++ [[ -n 0 ]]", "+++ heat-manage db_sync", "++++ openstack domain list", "++++ grep heat", "++++ awk '{print $4}'", "+++ CURRENT_HEAT_DOMAIN_NAME='heat_user_domain", "heat'", "+++ [[ heat_user_domain != \\h\\e\\a\\t\\_\\u\\s\\e\\r\\_\\d\\o\\m\\a\\i\
\\", "\\h\\e\\a\\t ]]", "+++ openstack domain create heat_user_domain", "Conflict occurred attempting to store domain - Duplicate Entry (HTTP 409) (Request-ID: req-c2e725bb-ebb8-40ef-9563-77977a6b959a)"], "stdout": "2019-08-12 09:46:37.324 17 INFO migrate.versioning.api [-] 70 -> 71... \u001b[00m

The work around is to delete the heat_user_domain in Keystone.

Tags:

Joseignacio-carretero (joseignacio-carretero) on 2019-08-12

summary:

- ansible-kolla fails to deploy heat if the groue heat_user_domain already
+ ansible-kolla fails to deploy heat if the heat_user_domain already
exists in keystone

Revision history for this message

Joseignacio-carretero (joseignacio-carretero) wrote on 2019-08-12: Re: ansible-kolla fails to deploy heat if the heat_user_domain already exists in keystone

As a complementary information, this error also happen when Heat is deployed and "kolla-ansible -i inventory reconfigure" is run.

The work around is to remove the domain again:
openstack domain set --disable heat_user_domain
openstack domain delete heat_domain_admin

Revision history for this message

Mark Goddard (mgoddard) wrote on 2019-08-13:

This problem will keep happening because kolla uses the domain heat_user_domain domain. This is configured in heat.conf as stack_user_domain_name. If your old deployment used the domain 'heat' for heat, you need to delete that instead.

Clearly this could be more flexible in kolla/kolla-ansible, and allow the domain name to be configured.

summary:

- ansible-kolla fails to deploy heat if the heat_user_domain already
+ kolla-ansible fails to deploy heat if the heat_user_domain already
exists in keystone

Revision history for this message

Mark Goddard (mgoddard) wrote on 2019-08-13:

Just to clarify the bug description - the problem is not that heat_user_domain exists, it is that the heat domain exists.

Relevant code in kolla, docker/heat/heat-api/extend_start.sh:

#!/bin/bash

# Bootstrap and exit if KOLLA_BOOTSTRAP variable is set. This catches all cases
# of the KOLLA_BOOTSTRAP variable being set, including empty.
if [[ "${!KOLLA_BOOTSTRAP[@]}" ]]; then
heat-manage db_sync
CURRENT_HEAT_DOMAIN_NAME=$(openstack domain list | grep heat | awk '{print $4}')

    if [[ "heat_user_domain" != "$CURRENT_HEAT_DOMAIN_NAME" ]]; then
        openstack domain create heat_user_domain
        openstack user create --domain heat_user_domain heat_domain_admin --password ${HEAT_DOMAIN_ADMIN_PASSWORD}
        openstack role add --domain heat_user_domain --user-domain heat_user_domain --user heat_domain_admin admin
    fi
    exit 0
fi

Alternatively we could make the above match more strict, to match on heat_user_domain exactly.

Changed in kolla-ansible:
importance:	Undecided → Medium

Revision history for this message

Joseignacio-carretero (joseignacio-carretero) wrote on 2019-08-13:

Yes, when "heat_user_domain" exists and heat installation fails because of that. Both kolla-ansible deploy and kolla-ansible reconfigure are failing for that reason. I think it would be nice checking if the domain exists and if so, using the existent domain instead of trying to create it.

Revision history for this message

Mark Goddard (mgoddard) wrote on 2019-08-13:

The problem isn't that heat_user_domain exists. The problem is that heat_user_domain and another domain called heat both exist, which breaks our check to see if heat_user_domain exists.

Fabian Bermel (fbermel) on 2020-03-09

Changed in kolla-ansible:
assignee:	nobody → Fabian Bermel (fbermel)

Fabian Bermel (fbermel) on 2020-03-09

affects:	kolla-ansible → kolla
Changed in kolla:
assignee:	Fabian Bermel (fbermel) → nobody
assignee:	nobody → Fabian Bermel (fbermel)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-09: Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.opendev.org/711923

Changed in kolla:
status:	New → In Progress

OpenStack Infra (hudson-openstack) on 2020-10-04

Changed in kolla:
assignee:	Fabian Bermel (fbermel) → Radosław Piliszek (yoctozepto)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-10-05: Fix merged to kolla (master)

Reviewed: https://review.opendev.org/711923
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=5e946cac490077011709c7587dc5e0b06927464f
Submitter: Zuul
Branch: master

commit 5e946cac490077011709c7587dc5e0b06927464f
Author: Fabian Bermel <email address hidden>
Date: Mon Mar 9 14:42:34 2020 +0100

Checks for heat_user_domain explicitly

The script should check for heat_user_domain explicitly instead of matching on "heat"

Closes-Bug: #1839824
Change-Id: Iac83444487d2aa96a0f90086029f43922d1e7856

Changed in kolla:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-10-14: Fix proposed to kolla (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/758058

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-10-14: Fix proposed to kolla (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/758059

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-10-14: Fix proposed to kolla (stable/stein)

#10

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/758060

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-10-16: Fix merged to kolla (stable/train)

#11

Reviewed: https://review.opendev.org/758059
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=353d7cb279d91fe48a73ccdfcddf99f87f511316
Submitter: Zuul
Branch: stable/train

commit 353d7cb279d91fe48a73ccdfcddf99f87f511316
Author: Fabian Bermel <email address hidden>
Date: Mon Mar 9 14:42:34 2020 +0100

Checks for heat_user_domain explicitly

The script should check for heat_user_domain explicitly instead of matching on "heat"

    Closes-Bug: #1839824
    Change-Id: Iac83444487d2aa96a0f90086029f43922d1e7856
    (cherry picked from commit 5e946cac490077011709c7587dc5e0b06927464f)

tags:	added: in-stable-train
tags:	added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-10-16: Fix merged to kolla (stable/stein)

#12

Reviewed: https://review.opendev.org/758060
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=bb62d2559fda8431c6e9721ae6ca6b6d0755921c
Submitter: Zuul
Branch: stable/stein

commit bb62d2559fda8431c6e9721ae6ca6b6d0755921c
Author: Fabian Bermel <email address hidden>
Date: Mon Mar 9 14:42:34 2020 +0100

Checks for heat_user_domain explicitly

The script should check for heat_user_domain explicitly instead of matching on "heat"

    Closes-Bug: #1839824
    Change-Id: Iac83444487d2aa96a0f90086029f43922d1e7856
    (cherry picked from commit 5e946cac490077011709c7587dc5e0b06927464f)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-10-16: Fix merged to kolla (stable/ussuri)

#13

Reviewed: https://review.opendev.org/758058
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=027016e98f72102718b2a7b5215be6ea4f03725f
Submitter: Zuul
Branch: stable/ussuri

commit 027016e98f72102718b2a7b5215be6ea4f03725f
Author: Fabian Bermel <email address hidden>
Date: Mon Mar 9 14:42:34 2020 +0100

Checks for heat_user_domain explicitly

The script should check for heat_user_domain explicitly instead of matching on "heat"

    Closes-Bug: #1839824
    Change-Id: Iac83444487d2aa96a0f90086029f43922d1e7856
    (cherry picked from commit 5e946cac490077011709c7587dc5e0b06927464f)