when os-*-hostname override is used, apache2 fails to start due to unexpected SSL cert/key path

Bug #1711360 reported by Nobuto Murata
26
This bug affects 7 people
Affects Status Importance Assigned to Milestone
Charm Helpers
Fix Released
High
Unassigned
OpenStack Glance Charm
Invalid
High
Pen Gale

Bug Description

I deployed a multi-network OpenStack. Also I have a genuinely signed SSL certificate by GeoTrust and an equivalet key.

To use the SSL cert/key, I have setup an external DNS to return a global IP address with glance.example.com, and an internal DNS to return private IP address with the same hostname as glance.example.com.

My glance charm configurations look like:

      os-admin-hostname: glance.example.com
      os-admin-network: 10.10.20.0/22
      os-internal-hostname: glance.example.com
      os-internal-network: 10.10.20.0/22
      os-public-hostname: glance.example.com
      os-public-network: 103.X.Y.Z/26
      ssl_cert: (base64 encoded cert)
      ssl_key: (base64 encoded key)
      vip: 10.10.X.Y 103.X.Y.Z

With the configurations above, the charm writes cert/key into /etc/apache2/ssl/glance/ as follows:

$ ls -1 /etc/apache2/ssl/glance/
cert_glance.example.com
key_glance.example.com

However, /etc/apache2/sites-enabled/openstack_https_frontend.conf requires {cert,key}_IP_ADDRESS instead. Thus, Apache fails to start.

<VirtualHost 10.10.21.178:8766>
    SSLCertificateFile /etc/apache2/ssl/cinder/cert_10.10.20.175
    SSLCertificateChainFile /etc/apache2/ssl/cinder/cert_10.10.20.175
    SSLCertificateKeyFile /etc/apache2/ssl/cinder/key_10.10.20.175
...
<VirtualHost 103.X.Y.Z:8766>
    SSLCertificateFile /etc/apache2/ssl/cinder/cert_103.X.Y.Z
    SSLCertificateChainFile /etc/apache2/ssl/cinder/cert_X.Y.Z
    SSLCertificateKeyFile /etc/apache2/ssl/cinder/key_103.X.Y.Z

Revision history for this message
Nobuto Murata (nobuto) wrote :

Sorry, I mistakenly press "submit" while I was still writing...

Changed in charm-glance:
status: New → Incomplete
Revision history for this message
Nobuto Murata (nobuto) wrote :

Possibly a duplicate of bug 1522932. However, this is not a keystone charm bug, because keystone charm works with the equivalent configs. Filing a separate bug to make sure I'm not hijacking the other bug.

description: updated
Changed in charm-glance:
status: Incomplete → New
Revision history for this message
Nobuto Murata (nobuto) wrote :

Following the comment in https://bugs.launchpad.net/charms/+source/keystone/+bug/1522932/comments/3

I patched the charm-helper code in glance charm by replacing the canonical_names with the keystone charm one. Then it works for me.

$ git diff
diff --git a/charms/glance/charmhelpers/contrib/openstack/context.py b/charms/glance/charmhelpers/contrib/openstack/context.py
index 73314a8..8689651 100644
--- a/charms/glance/charmhelpers/contrib/openstack/context.py
+++ b/charms/glance/charmhelpers/contrib/openstack/context.py
@@ -682,15 +682,14 @@ class ApacheSSLContext(OSContextGenerator):
     def canonical_names(self):
         """Figure out which canonical names clients will access this service.
         """
- cns = []
- for r_id in relation_ids('identity-service'):
- for unit in related_units(r_id):
- rdata = relation_get(rid=r_id, unit=unit)
- for k in rdata:
- if k.startswith('ssl_key_'):
- cns.append(k.lstrip('ssl_key_'))
+ # Replaced with the same function in hooks/keystone_context.py
+ # LP: #1711360
+ addresses = self.get_network_addresses()
+ addrs = []
+ for address, endpoint in addresses:
+ addrs.append(endpoint)

- return sorted(list(set(cns)))
+ return list(set(addrs))

     def get_network_addresses(self):
         """For each network configured, return corresponding address and vip

Revision history for this message
Nobuto Murata (nobuto) wrote :

After patched:

$ ls -1 /etc/apache2/ssl/glance/
/etc/apache2/ssl/glance/cert_10.10.20.176
/etc/apache2/ssl/glance/cert_103.X.Y.Z
/etc/apache2/ssl/glance/key_10.10.20.176
/etc/apache2/ssl/glance/key_103.X.Y.Z

Changed in charm-helpers:
assignee: nobody → Alex Kavanagh (ajkavanagh)
Changed in charm-glance:
assignee: nobody → Alex Kavanagh (ajkavanagh)
milestone: none → 17.08
Changed in charm-helpers:
status: New → Triaged
Changed in charm-glance:
status: New → Triaged
Changed in charm-helpers:
importance: Undecided → High
Changed in charm-glance:
importance: Undecided → High
James Page (james-page)
Changed in charm-glance:
milestone: 17.08 → 17.11
Chris Gregan (cgregan)
tags: added: cpe-onsite
Chris Gregan (cgregan)
tags: added: cdo-qa-blocker
Pen Gale (pengale)
Changed in charm-helpers:
assignee: Alex Kavanagh (ajkavanagh) → petevg (petevg)
Changed in charm-glance:
assignee: Alex Kavanagh (ajkavanagh) → petevg (petevg)
Revision history for this message
Pen Gale (pengale) wrote :

This is an interesting bug!

Right now, we don't actually have the right data available in charmhelpers to link the apache config file to the certs that we write out, especially in the cases where we have multiple networks linked to a single canonical name.

The quick and clever solution I envisioned when I picked up this bug isn't going to work. I'm working on a more comprehensive fix.

Revision history for this message
Pen Gale (pengale) wrote :

I tried replicating this in serverstack, but was unable to do so. Both the ssl files and the apache config used the canonical host name to name things.

Here's what I did to set things up:

1) I created three virtual networks via the openstack networking tools, and attached them to the machines that I was going to deploy charms to.

2) I deployed a minimal set of charms: glance, with keystone, rabbitmq-server and percona-cluster.

3) I deployed glance and configured it like so:

juju config glance \
     ssl_cert="$( base64 juju-d0e9cf-petevg-13.openstacklocal.crt )" ssl_key="$( base64 juju-d0e9cf-petevg-13.openstacklocal.key )" \
     os-public-network=10.6.3.0/24 os-admin-network=10.6.3.0/24 os-internal-network=192.168.23.0/24 \
     os-admin-hostname=juju-d0e9cf-petevg-13.openstacklocal os-internal-hostname=juju-d0e9cf-petevg-13.openstacklocal \
     os-public-hostname=juju-d0e9cf-petevg-13.openstacklocal \
     vip='192.168.X.Y 10.6.4.X'

Apache successfully starts, with ssl cert filenames that match the names and paths in the apache config.

I think that the codepath in charmhelpers is slightly different when it writes out the ssl certs and when it writes the apache config. When writing the ssl certs, it gets a list of canonical names, and writes out matching certs. When it writes out the config, it starts with a list of IPs, and overrides the IP of the endpoint if it has a canonical name for it. If the latter path breaks due to an issue with DNS, then you would see the behavior in the bug: the ssl cert gets tagged with the canonical name, but the apache config references the IP.

I would double check the DNS setup in the environment, and see if fixing it fixes this bug. If not, please re-open, and I will dig further into the issue.

Changed in charm-helpers:
status: Triaged → Incomplete
Changed in charm-glance:
status: Triaged → Incomplete
Revision history for this message
Nobuto Murata (nobuto) wrote :

@Peter,

Thank you for the detailed test. I don't have a handy environment for the full deployment and multi networks at this moment (It was reproduced reliably in a customer environment). Let me try to create a minimal reproducer.

Revision history for this message
Pen Gale (pengale) wrote :

@Nobuto: thank you for following up!

Let me know if there's any other testing that you'd like me to do from my end -- I haven't automated my test as much as I'd like, but I can perform it fairly quickly at this point, and can try out different scenarios.

Revision history for this message
Pen Gale (pengale) wrote :

This bug is still tugging at the back of my brain. I tried breaking my dns, both by shutting down my serverstack dns helper, and by simply choosing a canonical name that wasn't mapped in my bastion. In both cases, apache started, and the files were written correctly to disk. "Correctly" means that they were labelled and referenced by the canonical name that I set in the config.

I also tried reverting a minor patch that I had made to context.py, and that did not make any difference, either.

It would make me feel better to have a repro case, because my theory about it being due to broken dns is looking increasingly unlikely. It doesn't look like this is trivial to repro in my environment, however.

Revision history for this message
Nobuto Murata (nobuto) wrote :

Ok, I finally prepared a clean test bed with MAAS including multi-network. It looks like the situation as LP: #1711422.

The original issue is reproducible with rev. 258 (d456104952168d3b1009a9fb929c554f2625387e), but not with rev. 259 (9ef990444f0fd45b8049a8c1a978e2a0629a6854). So it's highly likely it's fixed during 17.08 cycle.

Possibly related:
https://github.com/juju/charm-helpers/commit/0087b1bc7331d2f6f843b2a98a4423c0462108aa
https://github.com/juju/charm-helpers/commit/9bb0f946368c9274970b064a1bc7b4b1e99fd753

Revision history for this message
Nobuto Murata (nobuto) wrote :

test bundle I used for the record.

Revision history for this message
Nobuto Murata (nobuto) wrote :

The result with rev. 259.

# ll /etc/apache2/ssl/glance/
total 16
dr-xr-xr-x 2 root root 4096 Oct 12 12:48 ./
dr-xr-xr-x 3 root root 4096 Oct 12 12:48 ../
-r--r--r-- 1 root root 1017 Oct 12 12:48 cert_glance.example.com
-r--r--r-- 1 root root 1704 Oct 12 12:48 key_glance.example.com

[/etc/apache2/sites-enabled/openstack_https_frontend.conf]
Listen 9282
<VirtualHost 192.168.151.126:9282>
    ServerName glance.example.com
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/glance/cert_glance.example.com
    # See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
    SSLCertificateChainFile /etc/apache2/ssl/glance/cert_glance.example.com
    SSLCertificateKeyFile /etc/apache2/ssl/glance/key_glance.example.com
    ProxyPass / http://localhost:9272/
    ProxyPassReverse / http://localhost:9272/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<VirtualHost 192.168.152.122:9282>
    ServerName glance.example.com
    SSLEngine on
    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
    SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCertificateFile /etc/apache2/ssl/glance/cert_glance.example.com
    # See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8
    SSLCertificateChainFile /etc/apache2/ssl/glance/cert_glance.example.com
    SSLCertificateKeyFile /etc/apache2/ssl/glance/key_glance.example.com
    ProxyPass / http://localhost:9272/
    ProxyPassReverse / http://localhost:9272/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
<Location />
    Order allow,deny
    Allow from all
</Location>

Changed in charm-helpers:
assignee: Pete Vander Giessen (petevg) → nobody
status: Incomplete → Fix Released
Changed in charm-glance:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.