tripleo

TLS everywhere: the apache service wrongly requests certificates for all networks

Series rocky
Bug #1811207

Bug #1811207 reported by Juan Antonio Osorio Robles on 2019-01-10

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
tripleo	Fix Released	Critical	Harald Jensås	tripleo stein-2
Queens	Fix Released	Critical	Harald Jensås	tripleo stein-2
Rocky	Fix Released	Critical	Harald Jensås	tripleo stein-2

Bug Description

we currently do a loop to go through all the networks as follows:

  ApacheNetworks:
    type: OS::Heat::Value
    properties:
      value:
        # NOTE(jaosorior) Get unique network names to create
        # certificates for those. We skip the tenant network since
        # we don't need a certificate for that.
        - ctlplane
{%- for network in networks %}
  {%- if network.name_lower != 'tenant' %}
        - {{network.name_lower}}
  {%- endif %}
{%- endfor %}

In puppet/services/apache.j2.yaml we don't/cannot filter based on the role.networks in the jinja loop[1] creating the ApacheNetworks value since this file is rendered for per stack, not per-role.
The ApacheNetworks is used later to pass the data to certmonger for certificate generation[2]:

              generate_service_certificates: true
              apache::mod::ssl::ssl_ca: {get_param: InternalTLSCAFile}
              apache::mod::ssl::ssl_protocol: ['all', '-SSLv2', '-SSLv3', '-TLSv1']
              tripleo::certmonger::apache_dirs::certificate_dir: '/etc/pki/tls/certs/httpd'
              tripleo::certmonger::apache_dirs::key_dir: '/etc/pki/tls/private/httpd'
              apache_certificates_specs:
                map_merge:
                  repeat:
                    template:
                      httpd-NETWORK:
                        service_certificate: '/etc/pki/tls/certs/httpd/httpd-NETWORK.crt'
                        service_key: '/etc/pki/tls/private/httpd/httpd-NETWORK.key'
                        hostname: "%{hiera('fqdn_NETWORK')}"
                        principal: "HTTP/%{hiera('fqdn_NETWORK')}"
                    for_each:
                        NETWORK: {get_attr: [ApacheNetworks, value]}

This results, in deployments with the default network setup, with certmonger trying to request a certificate for the management interface, and failing:

Request ID 'httpd-management':
        status: CA_REJECTED
        ca-error: Server at https://ipa.ooo.test/ipa/xml denied our request, giving up: 3007 (RPC failed at server. 'fqdn' is required).
        stuck: yes
        key pair storage: type=FILE,location='/etc/pki/tls/private/httpd/httpd-management.key'
        certificate: type=FILE,location='/etc/pki/tls/certs/httpd/httpd-management.crt'
        CA: IPA
        issuer:
        subject:
        expires: unknown
        pre-save command:
        post-save command: "systemctl reload httpd"
        track: yes
        auto-renew: yes

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/apache.j2.yaml#L65-L69
[2] https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/apache.j2.yaml#L106-L122

Juan Antonio Osorio Robles (juan-osorio-robles) on 2019-01-10

Changed in tripleo:
status:	New → Confirmed
importance:	Undecided → Critical
milestone:	none → stein-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-01-10: Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.openstack.org/629835

Changed in tripleo:
assignee:	nobody → Harald Jensås (harald-jensas)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-01-10: Fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/629835
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=a3aeb0b7faa6a817d7336d7fef86dd96bcf12731
Submitter: Zuul
Branch: master

commit a3aeb0b7faa6a817d7336d7fef86dd96bcf12731
Author: Harald Jensås <email address hidden>
Date: Thu Jan 10 12:22:46 2019 +0100

Filter apache_certificates_specs if hostname is empty

    Not all roles have all networks, when creating certificates
    we need to filter out certificate specs where the hiera
    lookup returns 'nil'.

    Previously we created noop resources with fallback to the
    ctlplane network for all networks defined in network data.
    So the hiera lookup returned a hostname for all networks
    no matter if the roles data included that network. This is
    no longer the case, so we need to filter.

Bonus, we no longer create certificates that are'nt actually
used.

Change-Id: I651919488cb68b0b9878b4e21ab376bfc6e3f0fe
Closes-Bug: #1811207

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-01-11: Fix proposed to puppet-tripleo (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/629961

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-01-11: Fix proposed to puppet-tripleo (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/629962

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-01-11: Fix merged to puppet-tripleo (stable/rocky)

Reviewed: https://review.openstack.org/629961
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=60a0f973fdfc9253fcdb8a335668d89f203a5d27
Submitter: Zuul
Branch: stable/rocky

commit 60a0f973fdfc9253fcdb8a335668d89f203a5d27
Author: Harald Jensås <email address hidden>
Date: Thu Jan 10 12:22:46 2019 +0100

Filter apache_certificates_specs if hostname is empty

    Not all roles have all networks, when creating certificates
    we need to filter out certificate specs where the hiera
    lookup returns 'nil'.

Bonus, we no longer create certificates that are'nt actually
used.

    Change-Id: I651919488cb68b0b9878b4e21ab376bfc6e3f0fe
    Closes-Bug: #1811207
    (cherry picked from commit a3aeb0b7faa6a817d7336d7fef86dd96bcf12731)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-01-11: Fix merged to puppet-tripleo (stable/queens)

Reviewed: https://review.openstack.org/629962
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=adfca27bd0d3ccdea8b57541a34c3cd13a7e1bb1
Submitter: Zuul
Branch: stable/queens

commit adfca27bd0d3ccdea8b57541a34c3cd13a7e1bb1
Author: Harald Jensås <email address hidden>
Date: Thu Jan 10 12:22:46 2019 +0100

Filter apache_certificates_specs if hostname is empty

    Not all roles have all networks, when creating certificates
    we need to filter out certificate specs where the hiera
    lookup returns 'nil'.

Bonus, we no longer create certificates that are'nt actually
used.

    Change-Id: I651919488cb68b0b9878b4e21ab376bfc6e3f0fe
    Closes-Bug: #1811207
    (cherry picked from commit a3aeb0b7faa6a817d7336d7fef86dd96bcf12731)