Heat stack with "wait_condition" fails with stack creation error

Could you please provide an output of heat stack-show command?

Looks like it affects this build too: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.services_ha_one_controller/24/testReport/(root)/deploy_sahara_ha_one_controller_tun/

From diagnostic snapshot logs of sahara it's clear that stack was always in CREATE_IN_PROGRESS state. Additionally, in 9.x sahara always creates wait conditions for all nodes in cluster, so if wait condition tests are not really working, that absolutely will affects sahara since we are consuming that feature.

[0] https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.services_ha_one_controller/24/testReport/(root)/deploy_sahara_ha_one_controller_tun/

I think it is problems with ssl. You should use correct templates.
https://review.openstack.org/#/c/330603/

we have some fix for ssl deployments
https://review.openstack.org/#/c/352416/
But at this moment a see some trouble in CI associated with the deployment environment:

https://custom-ci.infra.mirantis.net/view/All/job/9.x.custom.ubuntu.system_test/5/

https://custom-ci.infra.mirantis.net/view/All/job/9.x.custom.ubuntu.system_test/24/

Added tag "swarm-blocker" because it affects 3 swarm system tests (one heat, two sahara tests)

Fixes are on review:

https://review.openstack.org/#/q/Iaa0f4549586e3c421e3b7fe4573a61e0131d2e9e,n,z

we have some problem with create networks command in template.
Heat team are working on it

seems like we don't need to add custom -k argument to test, but add insecure=True to [clients_heat] section of heat.conf. So seems heat is misconfigured and should be fixed with puppets by adding insecure option during configurating, i.e.

[clients_heat]
insecure=true
url=...

I will write puppet team about that

I'm sure that it's not a good idea to use insecure for heat client for SSL case.

Denis, let me explain it a bit.

This option in heat.conf was added just for supporting easy work with WaitConditions.
It does not affect other security in Heat itself.

It was added in upstream for handling follow situation. User has template, which works on openstack without SSL.
Then he wants to use SSL or (copy example of template from somewhere, where was not used SSL).
He try to create stack with this template and got some unexpected error, due to internal heat misunderstanding.

How user should deal with it before?
there are two options:
1. add manually --insecure option in template (p.s. forgot to say, that config options insecure just trigger adding this option to WaitConditionHandle resource)

2. use full approach with certificates like generate them - upload one in Heat template and another to controller with heat services. (Honestly I have not heard that somebody followed this way. Most part prefer option #1)

So now we have option 3:
enable insecure WHEN we have ENABLE SSL (please make it dependence from SSL option, don't set it true for all - it's not bad, but looks weird :) )
So this bug is about option 3.

My personal preference is -k to the template to minimize risk but I'm not sure why the host it's running on does not have the proper CA in the system for validation. Where is the curl command ultimately run and are we not publishing the self signed CA everywhere? Or is this a custom SSL where the user does not provide a CA bundle? Adding insecure=True disables ssl certificate verification so while the traffic is still encrypted it does not prevent MITM attacks

Alex,
"Everything that is done is done for the best".

I re-reviewed all Heat code again + patch and unfortunately for me need to say,
that we should not change it.
This options also changes parameter for internal clients, that is not security safe way.
So generally you are right.

However all words about wait condition curl are true.
>> Where is the curl command ultimately run and are we not publishing the self signed CA everywhere?

This curl request will be executed in VM, which will be launched in Heat stack, i.e. usual VM booted by Nova. So issue here, that user should put certificate for Openstack inside his custom/own VM. Obviously He has not this certificate - because it's not security safe too.

I thought, that it's good solution for us, but unfortunately it just re-use existing "insecure" option, which is not what we want to have.

We will fix tests.

Hi! This commit has broken sahara https://github.com/openstack/heat/commit/59fc53a66c4dec45e4d150bce0a1d674477f710c and WC(if we correctly understand it). In logs of instance we can see:

Traceback (most recent call last):
  File "/var/lib/cloud/instance/scripts/loguserdata.py", line 32, in <module>
    import pkg_resources
ImportError: No module named 'pkg_resources'

When we revert this commit everything works fine.

For reproduce this issue you can create stack with one instance(with WC) and open logs of instance.

https://review.openstack.org/#/c/330603/ in master has been merged.
cherry-pick for mitaka waiting in the wings https://review.openstack.org/#/c/352416/

This bug is not a blocker anymore, cause two affected Sahara tests were fixed by other patch within another related bug.

Fix merged in stable/mitaca
https://review.openstack.org/#/c/352416/4

Swarm tests are green.
https://product-ci.infra.mirantis.net/view/9.x/job/9.x.system_test.ubuntu.services_ha_one_controller/53/
On my env 9.1 this test working fine