when os-*-hostname override is used, apache2 fails to start due to unexpected SSL cert/key path

Sorry, I mistakenly press "submit" while I was still writing...

Possibly a duplicate of bug 1522932. However, this is not a keystone charm bug, because keystone charm works with the equivalent configs. Filing a separate bug to make sure I'm not hijacking the other bug.

Following the comment in https://bugs.launchpad.net/charms/+source/keystone/+bug/1522932/comments/3

I patched the charm-helper code in glance charm by replacing the canonical_names with the keystone charm one. Then it works for me.

$ git diff
diff --git a/charms/glance/charmhelpers/contrib/openstack/context.py b/charms/glance/charmhelpers/contrib/openstack/context.py
index 73314a8..8689651 100644
--- a/charms/glance/charmhelpers/contrib/openstack/context.py
+++ b/charms/glance/charmhelpers/contrib/openstack/context.py
@@ -682,15 +682,14 @@ class ApacheSSLContext(OSContextGenerator):
     def canonical_names(self):
         """Figure out which canonical names clients will access this service.
         """
- cns = []
- for r_id in relation_ids('identity-service'):
- for unit in related_units(r_id):
- rdata = relation_get(rid=r_id, unit=unit)
- for k in rdata:
- if k.startswith('ssl_key_'):
- cns.append(k.lstrip('ssl_key_'))
+ # Replaced with the same function in hooks/keystone_context.py
+ # LP: #1711360
+ addresses = self.get_network_addresses()
+ addrs = []
+ for address, endpoint in addresses:
+ addrs.append(endpoint)

- return sorted(list(set(cns)))
+ return list(set(addrs))

     def get_network_addresses(self):
         """For each network configured, return corresponding address and vip

After patched:

$ ls -1 /etc/apache2/ssl/glance/
/etc/apache2/ssl/glance/cert_10.10.20.176
/etc/apache2/ssl/glance/cert_103.X.Y.Z
/etc/apache2/ssl/glance/key_10.10.20.176
/etc/apache2/ssl/glance/key_103.X.Y.Z

This is an interesting bug!

Right now, we don't actually have the right data available in charmhelpers to link the apache config file to the certs that we write out, especially in the cases where we have multiple networks linked to a single canonical name.

The quick and clever solution I envisioned when I picked up this bug isn't going to work. I'm working on a more comprehensive fix.

I tried replicating this in serverstack, but was unable to do so. Both the ssl files and the apache config used the canonical host name to name things.

Here's what I did to set things up:

1) I created three virtual networks via the openstack networking tools, and attached them to the machines that I was going to deploy charms to.

2) I deployed a minimal set of charms: glance, with keystone, rabbitmq-server and percona-cluster.

3) I deployed glance and configured it like so:

juju config glance \
     ssl_cert="$( base64 juju-d0e9cf-petevg-13.openstacklocal.crt )" ssl_key="$( base64 juju-d0e9cf-petevg-13.openstacklocal.key )" \
     os-public-network=10.6.3.0/24 os-admin-network=10.6.3.0/24 os-internal-network=192.168.23.0/24 \
     os-admin-hostname=juju-d0e9cf-petevg-13.openstacklocal os-internal-hostname=juju-d0e9cf-petevg-13.openstacklocal \
     os-public-hostname=juju-d0e9cf-petevg-13.openstacklocal \
     vip='192.168.X.Y 10.6.4.X'

Apache successfully starts, with ssl cert filenames that match the names and paths in the apache config.

I think that the codepath in charmhelpers is slightly different when it writes out the ssl certs and when it writes the apache config. When writing the ssl certs, it gets a list of canonical names, and writes out matching certs. When it writes out the config, it starts with a list of IPs, and overrides the IP of the endpoint if it has a canonical name for it. If the latter path breaks due to an issue with DNS, then you would see the behavior in the bug: the ssl cert gets tagged with the canonical name, but the apache config references the IP.

I would double check the DNS setup in the environment, and see if fixing it fixes this bug. If not, please re-open, and I will dig further into the issue.

@Peter,

Thank you for the detailed test. I don't have a handy environment for the full deployment and multi networks at this moment (It was reproduced reliably in a customer environment). Let me try to create a minimal reproducer.

@Nobuto: thank you for following up!

Let me know if there's any other testing that you'd like me to do from my end -- I haven't automated my test as much as I'd like, but I can perform it fairly quickly at this point, and can try out different scenarios.

This bug is still tugging at the back of my brain. I tried breaking my dns, both by shutting down my serverstack dns helper, and by simply choosing a canonical name that wasn't mapped in my bastion. In both cases, apache started, and the files were written correctly to disk. "Correctly" means that they were labelled and referenced by the canonical name that I set in the config.

I also tried reverting a minor patch that I had made to context.py, and that did not make any difference, either.

It would make me feel better to have a repro case, because my theory about it being due to broken dns is looking increasingly unlikely. It doesn't look like this is trivial to repro in my environment, however.

Ok, I finally prepared a clean test bed with MAAS including multi-network. It looks like the situation as LP: #1711422.

The original issue is reproducible with rev. 258 (d456104952168d3b1009a9fb929c554f2625387e), but not with rev. 259 (9ef990444f0fd45b8049a8c1a978e2a0629a6854). So it's highly likely it's fixed during 17.08 cycle.

Possibly related:
https://github.com/juju/charm-helpers/commit/0087b1bc7331d2f6f843b2a98a4423c0462108aa
https://github.com/juju/charm-helpers/commit/9bb0f946368c9274970b064a1bc7b4b1e99fd753

test bundle I used for the record.

The result with rev. 259.

# ll /etc/apache2/ssl/glance/
total 16
dr-xr-xr-x 2 root root 4096 Oct 12 12:48 ./
dr-xr-xr-x 3 root root 4096 Oct 12 12:48 ../
-r--r--r-- 1 root root 1017 Oct 12 12:48 cert_glance.example.com
-r--r--r-- 1 root root 1704 Oct 12 12:48 key_glance.example.com

[/etc/apache2/sites-enabled/openstack_https_frontend.conf]
Listen 9282
<VirtualHost 192.168.151.126:9282>
    ServerName glance.example.com
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/glance/cert_glance.example.com
    # See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
    SSLCertificateChainFile /etc/apache2/ssl/glance/cert_glance.example.com
    SSLCertificateKeyFile /etc/apache2/ssl/glance/key_glance.example.com
    ProxyPass / http://localhost:9272/
    ProxyPassReverse / http://localhost:9272/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<VirtualHost 192.168.152.122:9282>
    ServerName glance.example.com
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/glance/cert_glance.example.com
    # See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
    SSLCertificateChainFile /etc/apache2/ssl/glance/cert_glance.example.com
    SSLCertificateKeyFile /etc/apache2/ssl/glance/key_glance.example.com
    ProxyPass / http://localhost:9272/
    ProxyPassReverse / http://localhost:9272/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
<Location />
    Order allow,deny
    Allow from all
</Location>