Aodh with ssl

Bug #1578241 reported by Alexander Nagovitsyn
This bug report is a duplicate of:  Bug #1576520: Heat autoscaling does not work. Edit Remove
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
Confirmed
High
Peter Razumovsky

Bug Description

This bug is related to https://bugs.launchpad.net/fuel/+bug/1576520

Autoscaling of heat cluster does not work.
Seems that AODH does not handle alarms with ssl enviroment.
Most likely the root cause is ceilometer-->heat-cfn[public] alert rule is placed with HTTP protocol while HTTPS is required.

Aodh-notifier:
http://paste.openstack.org/show/496081/
http://paste.openstack.org/show/496077/

Alarm list:
http://paste.openstack.org/show/495714/

Steps: run Autoscaling test using ostf or using heat-template:
http://paste.openstack.org/show/496084/

Note:
alarms are updated every 10 minutes.

affects: fuel → mos
Changed in mos:
milestone: 9.0 → none
Vitaly Gusev (vgusev)
Changed in mos:
assignee: nobody → MOS Ceilometer (mos-ceilometer)
milestone: none → 9.0
Revision history for this message
Bug Checker Bot (bug-checker) wrote : Autochecker

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags: added: need-info
Changed in mos:
status: New → Confirmed
tags: added: area-ceilometer
removed: ceilometer
Ilya Tyaptin (ityaptin)
tags: added: area-heat
Revision history for this message
Ilya Tyaptin (ityaptin) wrote :

I've checked this error and found out that heat autoscaling rules create a alarms with http:// alarm hooks.
It may cause an issue in notification agent, because this url is not available via http, only with https.

Alarm show:
http://paste.openstack.org/show/496233/

tags: added: blocker-for-qa
Revision history for this message
Ilya Tyaptin (ityaptin) wrote :

During this bug researching next unexpected behaviors have been found:
1. Heat create an alarm action with http scheme instead of https
2. Heat create an alarm action with ip instead of hostname (public.fuel.local). It causes an issue because certificate is signed for the public.fuel.local, not ip. So, aodh alarm notifying fails with "Hostname not matching".
3. After alarm updating to the https scheme and valid hostname post request from aodh notifier returns "403 <ErrorResponse><Error><Message>User is not authorized to perform action</Message><Code>AccessDenied</Code><Type>Sender</Type></Error></ErrorResponse>"

Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

alarm urls created by Heat are ec2-signed, so changing them after they have been created is not an option as this invalidates them.

Instead (see [0]) we should pick the right combination of options in heat.conf and keystone catalog entries to make sure the url created by Heat can be accessed by aodh

It would seem, that picking the right options might involve some trade-offs still. E.g. we could force Heat to create alarm urls on private API endpoints, but then waitconditions will most probably not work (as there is no routing from tenant's network to control plane network) and by doing so we effectively would prohibit manual scaling from outside of the cloud. I remember I had some round-off of trade-offs somewhere prepped some time ago, will try to find it.

[0] https://github.com/openstack/heat/blob/master/heat/engine/resources/signal_responder.py#L146-L152

Revision history for this message
Ilya Tyaptin (ityaptin) wrote :

In my environment, after changing heat config option "heat_waitcondition_server_url" from the format `http://<ip>:8000/v1/waitcondition` to `https://public.fuel.local:8000/v1/waitcondition` fixes issue.

Revision history for this message
Peter Razumovsky (prazumovsky) wrote :

me too

Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

just remember, that for this to work with waitconditions, the tenant's network _must_ have working DNS to resolve the "public.fuel.local" domain name - and IMO that's not something that should be really taken for granted. We need to double-check this - create a new network/subnet in Neutron, do not do anything re DNS there and try booting a stack with VM in this new subnet and waitcondition.

Revision history for this message
Oleksii Chuprykov (ochuprykov) wrote :

Pavlo, I've verified your case. The problem is that mos 9.0 + ssl doesn't work with waitconditions by default now. In order to make it working, we need to put url=https://<ip>:8004/v1/%(tenant_id)s in [clienst_heat] section of heat.conf. After that waitconditions will work. heat_waitcondition_server_url doesn't influence behaviour of waitconditions (this is strange, probably this is because of particular configs), so this solution seems acceptable to me.

Changed in mos:
assignee: MOS Ceilometer (mos-ceilometer) → MOS Heat (mos-heat)
Revision history for this message
Peter Razumovsky (prazumovsky) wrote :

Next updates:

1. Currently aodh cannot handle case, when we give it url with ip (aodh certificate signed with domain);

2. Waitcondition cannot work, when we give it url with domain.

So, currently there is only one solution: extra option url for alarm url. This url will be build with https scheme and domain. waitcondition url will be build with https scheme and ip.

Revision history for this message
Peter Razumovsky (prazumovsky) wrote :

Possible solution at this moment is:

* define default config of heat_waitcondition_server_url with https scheme and domain instead of ip (only waitcondition); (this option will be used by aodh)

* define default config in section [clients_heat] url = https://<ip>:8004/v1/%(tenant_id)s. (this option will be used by waitcondition)

Changed in mos:
assignee: MOS Heat (mos-heat) → Peter Razumovsky (prazumovsky)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.