heat-container-agent fails to communicate with Keystone.

Bug #1744362 reported by Syed Armani
38
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Magnum
Confirmed
Medium
Unassigned

Bug Description

OpenStack release: stable/pike
Host OS: Ubuntu 16.04 LTS
Deployment method: OSA playbooks
Glance image used for K8 cluster: Fedora

Kubernetes master node remain stuck in "CREATE_IN_PROGRESS" state because the resource "enable_prometheus_monitoring_deployment" never completes its run.

# openstack stack resource list --> http://paste.openstack.org/show/647007/

# systemctl status heat-container-agent --> http://paste.openstack.org/show/647025/
-- Authorization failed: Unable to establish connection
-- No local metadata found
-- /var/lib/os-collect-config/local-data not found.

The /etc/os-collect-config.conf file in the kubernetes master also contains the Keystone internal endpoint.

Revision history for this message
yatin (yatinkarel) wrote :

We should check and support configuring deploy_auth_url to public
Just for reference deploy_auth_url is being taken from here:- https://github.com/openstack/heat/blob/bdddeee60212627b3beed41ca5ef5be56fd05b50/heat/common/context.py#L222-L233

Changed in magnum:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Christian Zunker (christian-zunker) wrote :

I had the same problem with a k8s cluster.

These two helped me fix the problem:
https://ask.openstack.org/en/question/102214/software-deployment-in-heat-problem-with-os-collect-config/
https://bugs.launchpad.net/kolla-ansible/+bug/1762754

The problem was actually in heat config, not magnum.

Revision history for this message
Mohammed Naser (mnaser) wrote :
Revision history for this message
Saibal Dey (saibaldey) wrote :

The root cause of the issue is hostname/DNS resolution fails for auth-URLs & few other service URLS (magnum,heat etc).
Generally while configuring the OpenStack services we use "controller" instead of the IP, like example:

For Keystone:
keystone-manage bootstrap --bootstrap-password ADMIN_PASS \
  --bootstrap-admin-url http://controller:5000/v3/ \
  --bootstrap-internal-url http://controller:5000/v3/ \
  --bootstrap-public-url http://controller:5000/v3/ \
  --bootstrap-region-id RegionOne

For Magnum:
openstack endpoint create --region RegionOne \
  container-infra public http://controller:9511/v1

So when the k8s master node gets provisioned (as VM) it creates couple of containers, "heat-container-agent" which interns use those able mentioned configs to get the k8s registered & configured with the OpenStack. As the container can't resolve the "controller" DNS entry, so all communications with OpenStack API fails. So there are couple of fixes for this:
1. Configure an internal DNS so resolute the "controller" URLs or
2. Use IP instead of "controller" in those able mentioned configs.

Option 2 is for POC & #1 should be considered for the production or HA OpenStack clusters.

Revision history for this message
Satish Patel (satish-txt) wrote :

I have same issue but on different heat (during kube master deployment it stuck in CREATE_IN_PROGRESS).

In my case i have HAProxy running on privet IP range which is not routable and magnum kubernetes trying to deploy master and its trying to talk to keystone and failing because its not on public network where it can talk to keystone. Is there a way to disable keystone dependency from kubernetes?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.