Heat service returns Internal Server Error when enabling kolla_enable_tls_external

Bug #1812864 reported by Magnus Lööf
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Radosław Piliszek
Stein
Triaged
Medium
Radosław Piliszek
Train
Fix Released
Medium
Radosław Piliszek
Ussuri
Fix Released
Medium
Radosław Piliszek

Bug Description

When enabling parameter `kolla_enable_tls_external` and deploying Openstack using Kolla, Heat Orchestration does not work properly.

- Accessing any of the menus under Orchestration in Horizon GUI over TLS fails, all other parts of GUI works
- Accessing Heat API over public interface with TLS fails with "Internal Error"
- Accessing Heat API over internal interface fails with "Internal Error"

Error logs from heat api

https://pastebin.com/445AjK2L

The problem can be resolved by reverting this change: https://review.openstack.org/#/c/566361/

The reason the problem appears are:

- The TLS certificate is not trusted by the container, if a private CA not in ca-certificates package is used
- The external FQDN cannot be reached from inside the internal network. For security reasons, this can be assumed to be required.

Revision history for this message
Jesús Arias Gil (jesusarias95) wrote :

Hi, any solution for this bug?

Revision history for this message
Magnus Lööf (magnus-loof) wrote :

I did a workaround and put this in the config files for Heat:

```
{% if openstack_enable_tls_external %}
[clients_keystone]
auth_uri = http://{{ openstack_internal_vip }}:5000
{% endif %}
```

Revision history for this message
Mark Goddard (mgoddard) wrote :

The reason for adding https://review.openstack.org/#/c/566361/ is that it is passed to instances by heat for calling back to the heat API. Instances can't be assumed to have access to the internal API, so we need to pass the external URL.

I can see why this causes issues with external TLS enabled however, since the container won't necessarily have the necessary certificates trusted. The workaround posted above should work.

Revision history for this message
Mark Goddard (mgoddard) wrote :

Would it be helpful to provide a flag in kolla-ansible that sets whether to use the internal or external keystone endpoint in this config section?

Revision history for this message
Mark Goddard (mgoddard) wrote :

It would be good for heat to provide a separate configuration option for its own access to the keystone endpoint vs that which is passed to instances with software deployments.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/676716

Changed in kolla-ansible:
assignee: nobody → Mark Goddard (mgoddard)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/676716
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=d54c8fbdccfd8145b65d0fcc50dc8628b37c1a88
Submitter: Zuul
Branch: master

commit d54c8fbdccfd8145b65d0fcc50dc8628b37c1a88
Author: Mark Goddard <email address hidden>
Date: Thu Aug 15 16:04:44 2019 +0100

    Use internal API for heat -> heat communication

    Heat has a new option (server_keystone_endpoint_type), which can be used
    to set the keystone endpoint used by instances to make callbacks to
    heat. This needs to be public, since we can't assume users have access
    to the internal API. However, the current method of setting
    [clients_heat] endpoint_type means that communication from heat to its
    own API (e.g. when a stack is a resource in another stack) uses the
    public network also, and this might not work if TLS is enabled.

    This change uses server_keystone_endpoint_type to keep instance traffic
    on the public API, and removes the [clients_heat] endpoint_type option
    to use the default in [clients] endpoint_type of internalURL.

    This feature was added to heat in https://review.opendev.org/#/c/650967.

    Change-Id: I932ea55a3c2a411557c34361db08bcb3a2b27eaf
    Closes-Bug: #1812864
    Related-Bug: #1762754
    Related-Bug: #1688331

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 9.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Mark, this is not fixed. Heat still tries keystone over public endpoint. Only heat-to-heat was fixed.

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Also, the feature was backported to stein, just not released. Negotiating new heat release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/694666

Changed in kolla-ansible:
assignee: Mark Goddard (mgoddard) → Radosław Piliszek (yoctozepto)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/694666
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=2cd00dadc0ad06610e94f879ddce145ad1134ea5
Submitter: Zuul
Branch: master

commit 2cd00dadc0ad06610e94f879ddce145ad1134ea5
Author: Radosław Piliszek <email address hidden>
Date: Sat Nov 16 12:30:46 2019 +0100

    Use internal API for heat -> keystone communication

    Continues work from https://review.opendev.org/676716

    Change-Id: If0195c38034d404849bf2e8fca4629b2d38a2680
    Closes-Bug: #1812864
    Related-Bug: #1762754
    Related-Bug: #1688331

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/694985

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/694986

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (stable/stein)

Change abandoned by Radosław Piliszek (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/694986
Reason: not now

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/694985
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=21e5924ef4f20e5e39bd708388088e565936e868
Submitter: Zuul
Branch: stable/train

commit 21e5924ef4f20e5e39bd708388088e565936e868
Author: Radosław Piliszek <email address hidden>
Date: Sat Nov 16 12:30:46 2019 +0100

    Use internal API for heat -> keystone communication

    Continues work from https://review.opendev.org/676716

    Change-Id: If0195c38034d404849bf2e8fca4629b2d38a2680
    Closes-Bug: #1812864
    Related-Bug: #1762754
    Related-Bug: #1688331
    (cherry picked from commit 2cd00dadc0ad06610e94f879ddce145ad1134ea5)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 9.0.0.0rc2

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc2 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.