apache2 fails to start due to missing certificate and key

Subscribed to Field Critical SLA, as this bug will cause all SSL enabled deployments of gnocchi to fail.

The issue can be reproduced with the following deployment. Crashdump is attached.

juju status: https://pastebin.canonical.com/p/MmR9nRdsrz/
bundle: https://pastebin.canonical.com/p/rC4yyKhW6j/

overlay-hostnames.yaml: https://pastebin.canonical.com/p/fszx8HTGQ6/
overlay-ssl.yaml: https://pastebin.canonical.com/p/YptYYc7XDw/
overlay-mysql-noha.yaml: https://pastebin.canonical.com/p/938fM5xn6q/

deploy command:
juju deploy -m foundations-maas:admin/ssl ./bundle.yaml --overlay ./overlay-hostnames.yaml --overlay ./overlay-ssl.yaml --overlay ./overlay-mysql-noha.yaml

Fix proposed to branch: master
Review: https://review.openstack.org/548067

Ashley,

Can you please do a test run with cs:~thedac/gnocchi-1? I think this resolves the Services not running apache2 problem. We may find other issues, but I am interested if we get past that one.

Performed some live testing during deploy time that highlights the race condition.

After, the first gnocchi unit advertises itself as ready to the ceilometer the other units are still in progress. The SSL version of the test shows this quite clearly. Hitting the VIP which then gets sent to HAProxy and an indeterminate gnocchi back end, some back ends are ready and respond and some have not even setup SSL yet:

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:46:47-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
WARNING: cannot verify 10.5.150.191's certificate, issued by ‘CN=Ubuntu Cloud Intermediate Certificate Authority,OU=Ubuntu Cloud,O=Ubuntu’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 144 [application/json]
Unable to establish SSL connection.

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:49:03-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
Saving to: ‘STDOUT’

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:49:22-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Unable to establish SSL connection.

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:49:24-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
Saving to: ‘STDOUT’

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:49:44-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
WARNING: cannot verify 10.5.150.191's certificate, issued by ‘CN=Ubuntu Cloud Intermediate Certificate Authority,OU=Ubuntu Cloud,O=Ubuntu’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 144 [application/json]
Saving to: ‘STDOUT’

2018-02-26 10:49:46 (11.4 MB/s) - written to stdout [144/144]

$ wget --no-check-certificate -O- https://10.5.150.191:8041
--2018-02-26 10:49:48-- https://10.5.150.191:8041/
Connecting to 10.5.150.191:8041... connected.
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Unable to esta...

@David - does cs:~thedac/gnocchi-1 includes the fix for 1749280 ?

Ashley, it has the updates to help mitigate, yes. It is built on the following:

https://review.openstack.org/#/c/545491/
https://review.openstack.org/#/c/548067/

The charm is applied, we will get some runs with the updated charm tonight.

I think the bug could be generalised to "API service advertises endpoint before all units of service are ready" and it applies to most (all ?) of our api charms I expect. So, I think option 2 is probably the correct one but I would say option 1 is fine for this particular instance. I am not keen on option 3. I think the contract between charms should be such that if a charm advertises and endpoint it is declaring that that endpoint is ready to use.

As I say, option 2 probably needs to be done regardless of whether option 1 is implemented. thedac has already done a lot of work in various charms to delay endpoint advertising. I think that work needs to be applied here with the additional delay of "If ssl is enabled don't advertise
endpoint until certs/keys are configured and installed on all units".

I'd agree with Liam's analysis - if we want to get a release out in the short-term, we'll need to go with 1) for now and work towards 2) over the next cycle; its a common pattern so needs careful design and implementation so not one to try and squeeze in after feature freeze.

We see the same error with gnocchi charm pointed to cs:~thedac/gnocchi-1. bundle.yaml and crashdump are attached.

Reviewed: https://review.openstack.org/548067
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=b0a1551e56f2a540942cbd745fa4658057100c5e
Submitter: Zuul
Branch: master

commit b0a1551e56f2a540942cbd745fa4658057100c5e
Author: David Ames <email address hidden>
Date: Mon Feb 26 08:55:14 2018 -0800

    Explicitly call configure_ssl

    Although, layer_openstack_api calls configure_ssl the ordering is
    indeterminate. Explicitly call configure_ssl in the charm layer before
    services are enabled to avoid attempting to start apache2 before
    certificates are available.

    Change-Id: Icb40bbcf4ca920be202584e9e25ffd1c2b9c8d61
    Partial-Bug: #1750915

Still seeing this issue quite often even with fixes in place. This will continue to be a field blocker at this occurrence rate

attached a crashdump from our most recent reproduction.

bundle:
http://paste.ubuntu.com/p/hNfNH4xHhV/

The bundle in comment #15 is not a reproducer. It appears to be missing overlays as provided earlier. Please confirm if/whether those have changed or are the same, preferrably by providing the complete stack of bundles with each run. Thank you.

here are the requested overlays from that run: http://paste.ubuntu.com/p/ySBtQYX6xb/

Reviewed: https://review.openstack.org/550104
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=4949dc24127a0de34195841f855b909f3ccf13ca
Submitter: Zuul
Branch: master

commit 4949dc24127a0de34195841f855b909f3ccf13ca
Author: Liam Young <email address hidden>
Date: Tue Mar 6 14:54:50 2018 +0000

    Render certs for all endpoint types

    When using user supplied certs directly to the charms make sure
    that the bundles which are passed in are rendered for all
    supported endpoint types (internal, admin and public).

    Closes-Bug: #1750915
    Change-Id: If62fd5528470653586b2be3aaef7f829e5abad40

Fix proposed to branch: master
Review: https://review.openstack.org/550120

Reviewed: https://review.openstack.org/550120
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=e3f0707e6a5af8451a13b21f43ac50f361a042f9
Submitter: Zuul
Branch: master

commit e3f0707e6a5af8451a13b21f43ac50f361a042f9
Author: David Ames <email address hidden>
Date: Tue Mar 6 16:27:46 2018 +0100

    Rebuild to pull in charms.openstack

    Pull in the fix for internal, public and admin certificates in
    charms.openstack.

    Change-Id: Ia6f6ee555f611da552337826f6fc6c87e34c7e55
    Closes-Bug: #1750915

Reviewed: https://review.openstack.org/550113
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=ce1e84794af48eb65266ab316830c50b3e2a2f25
Submitter: Zuul
Branch: master

commit ce1e84794af48eb65266ab316830c50b3e2a2f25
Author: David Ames <email address hidden>
Date: Tue Mar 6 16:25:43 2018 +0100

    Rebuild to pull in charms.openstack

    Pull in the fix for internal, public and admin certificates in
    charms.openstack.

    Change-Id: I40385c436d928e996239053f9103dc66b38f3e0f
    Closes-Bug: #1750915

Reviewed: https://review.openstack.org/550117
Committed: https://git.openstack.org/cgit/openstack/charm-designate/commit/?id=49c5fdeb828dfba647375d9effc82b9a3ccfbae1
Submitter: Zuul
Branch: master

commit 49c5fdeb828dfba647375d9effc82b9a3ccfbae1
Author: David Ames <email address hidden>
Date: Tue Mar 6 16:27:08 2018 +0100

    Rebuild to pull in charms.openstack

    Pull in the fix for internal, public and admin certificates in
    charms.openstack.

    Change-Id: I287151ede2156b58b3057d8d569a3298acee8664
    Closes-Bug: #1750915

https://solutions.qa.canonical.com/#/qa/testRun/9b7e6a95-ee90-4096-8ed6-480de5ff68b4

The patch fixed the apache2 issue. ceilometer still failed with the following error.

2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 288, in get
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/gnocchiclient/client.py", line 35, in request
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer **kwargs)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 192, in request
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer return self.session.request(url, method, **kwargs)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer return wrapped(*args, **kwargs)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 578, in request
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer auth_headers = self.get_auth_headers(auth)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 905, in get_auth_headers
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer return auth.get_headers(self, **kwargs)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 90, in get_headers
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer token = self.get_token(session)
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 89, in get_token
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer return self.get_access(session).auth_token
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 190939 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 135, in get_access
2018-03-07 14:48:00 DEBUG metric-service-relation-changed 2018-03-07 14:48:00.007 ...