neutron api tls issues

Bug #1852779 reported by Michele Baldessari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Michele Baldessari

Bug Description

There are two tls issues currently in neutron:
A) neutron_api is doing bind mounting tls certs directly from the host and then chowning them from then container.
"Source": "/etc/pki/tls/private/ovn_neutron_client.key",
"Destination": "/etc/pki/tls/private/ovn_neutron_client.key",

"Source": "/etc/pki/tls/certs/ovn_neutron_client.crt",
"Destination": "/etc/pki/tls/certs/ovn_neutron_client.crt",

And then it thinks it can chown it inside the container via the kolla script:
[root@controller-0 stdouts]# more /var/lib/kolla/config_files/neutron_api.json
...
        {
            "optional": true,
            "owner": "neutron:neutron",
            "path": "/etc/pki/tls/certs/ovn_neutron_client.crt",
            "perm": "0644"
        },
        {
            "optional": true,
            "owner": "neutron:neutron",
            "path": "/etc/pki/tls/private/ovn_neutron_client.key",
            "perm": "0644"
        }

What needs to happen here is that we drop the bind mounts, and copy in the certs via kolla and put them in /var/lib/config-data/puppet-generated/...... Just like galera is doing.

This typically breaks the updating of the certs, but that is currently not working for neutron_ovn anyway since the cert has no post-command in certmonger:
Request ID 'neutron_ovn':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/pki/tls/private/ovn_neutron_client.key'
        certificate: type=FILE,location='/etc/pki/tls/certs/ovn_neutron_client.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=REDHAT.LOCAL
        subject: CN=controller-1.internalapi.redhat.local,O=REDHAT.LOCAL
        expires: 2021-11-14 15:44:20 UTC
        dns: controller-1.internalapi.redhat.local
        principal name: <email address hidden>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes

Also fun fact, it would not even work with bind mounts because a new cert would create a new inode and when you bind mount a single file and change the inode on the host the container will not see any changes at all.

B) The logging in the neutron_server_tls_proxy container is not persisted to the host

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/694614

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Michele Baldessari (michele) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/694769

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/train)

Change abandoned by Kamil Sambor (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/694769

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/694614
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a7db39af7cf779e5e236c6f9b54c260feba24b98
Submitter: Zuul
Branch: master

commit a7db39af7cf779e5e236c6f9b54c260feba24b98
Author: Michele Baldessari <email address hidden>
Date: Fri Nov 15 18:50:28 2019 +0100

    Fix neutron api TLS issues

    There are two tls issues currently in neutron:
    A) neutron_api is doing bind mounting tls certs directly from the host and then chowning them from then container.
    "Source": "/etc/pki/tls/private/ovn_neutron_client.key",
    "Destination": "/etc/pki/tls/private/ovn_neutron_client.key",

    "Source": "/etc/pki/tls/certs/ovn_neutron_client.crt",
    "Destination": "/etc/pki/tls/certs/ovn_neutron_client.crt",

    And then it thinks it can chown it inside the container via the kolla script:
    [root@controller-0 stdouts]# more /var/lib/kolla/config_files/neutron_api.json
    ...
            {
                "optional": true,
                "owner": "neutron:neutron",
                "path": "/etc/pki/tls/certs/ovn_neutron_client.crt",
                "perm": "0644"
            },
            {
                "optional": true,
                "owner": "neutron:neutron",
                "path": "/etc/pki/tls/private/ovn_neutron_client.key",
                "perm": "0644"
            }

    What needs to happen here is that we drop the direct bind mounts, and copy in the certs via kolla. Just like galera is doing.

    Tested and with this patch I correctly get a Train/OSP16 TLS everywhere
    deployment with a working neutron:
    Before:
    (overcloud) [stack@undercloud-0 ~]$ openstack network list
    HttpException: 503: Server Error for url: https://overcloud.redhat.local:13696/v2.0/networks, No server is available to handle this request.: 503 Service Unavailable

    After:
    (overcloud) [stack@undercloud-0 ~]$ openstack network list
    (overcloud) [stack@undercloud-0 ~]$

    B) The logging in the neutron_server_tls_proxy container is not persisted to the host

    Before:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    [root@controller-0 ~]#

    After:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    total 136
    -rw-r--r--. 1 root root 400 Nov 15 17:43 error_log
    -rw-r--r--. 1 root root 72784 Nov 15 17:53 neutron-api-proxy_access_ssl.log
    -rw-r--r--. 1 root root 2748 Nov 15 17:43 neutron-api-proxy_error_ssl.log
    [root@controller-0 ~]#

    Change-Id: Ia1c22b8981fb735a052707516f73313c4e5ca93c
    Closes-Bug: #1852779

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/694769
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=491ffccbb99b6d6600e463991f20864cd1eb100e
Submitter: Zuul
Branch: stable/train

commit 491ffccbb99b6d6600e463991f20864cd1eb100e
Author: Michele Baldessari <email address hidden>
Date: Fri Nov 15 18:50:28 2019 +0100

    Fix neutron api TLS issues

    There are two tls issues currently in neutron:
    A) neutron_api is doing bind mounting tls certs directly from the host and then chowning them from then container.
    "Source": "/etc/pki/tls/private/ovn_neutron_client.key",
    "Destination": "/etc/pki/tls/private/ovn_neutron_client.key",

    "Source": "/etc/pki/tls/certs/ovn_neutron_client.crt",
    "Destination": "/etc/pki/tls/certs/ovn_neutron_client.crt",

    And then it thinks it can chown it inside the container via the kolla script:
    [root@controller-0 stdouts]# more /var/lib/kolla/config_files/neutron_api.json
    ...
            {
                "optional": true,
                "owner": "neutron:neutron",
                "path": "/etc/pki/tls/certs/ovn_neutron_client.crt",
                "perm": "0644"
            },
            {
                "optional": true,
                "owner": "neutron:neutron",
                "path": "/etc/pki/tls/private/ovn_neutron_client.key",
                "perm": "0644"
            }

    What needs to happen here is that we drop the direct bind mounts, and copy in the certs via kolla. Just like galera is doing.

    Tested and with this patch I correctly get a Train/OSP16 TLS everywhere
    deployment with a working neutron:
    Before:
    (overcloud) [stack@undercloud-0 ~]$ openstack network list
    HttpException: 503: Server Error for url: https://overcloud.redhat.local:13696/v2.0/networks, No server is available to handle this request.: 503 Service Unavailable

    After:
    (overcloud) [stack@undercloud-0 ~]$ openstack network list
    (overcloud) [stack@undercloud-0 ~]$

    B) The logging in the neutron_server_tls_proxy container is not persisted to the host

    Before:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    [root@controller-0 ~]#

    After:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    total 136
    -rw-r--r--. 1 root root 400 Nov 15 17:43 error_log
    -rw-r--r--. 1 root root 72784 Nov 15 17:53 neutron-api-proxy_access_ssl.log
    -rw-r--r--. 1 root root 2748 Nov 15 17:43 neutron-api-proxy_error_ssl.log
    [root@controller-0 ~]#

    Change-Id: Ia1c22b8981fb735a052707516f73313c4e5ca93c
    Closes-Bug: #1852779
    (cherry picked from commit a7db39af7cf779e5e236c6f9b54c260feba24b98)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 12.0.0

This issue was fixed in the openstack/tripleo-heat-templates 12.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/696652

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/696652
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0aecbc772384ae4623fe2013c1698825f268e16b
Submitter: Zuul
Branch: stable/stein

commit 0aecbc772384ae4623fe2013c1698825f268e16b
Author: Michele Baldessari <email address hidden>
Date: Fri Nov 15 18:50:28 2019 +0100

    Fix neutron api TLS logging

    This is a partial cherry pick (logging part) of
    Ia1c22b8981fb735a052707516f73313c4e5ca93c
    Closes-Bug: #1852779

    Namely it is the second part of the commit only:
    B) The logging in the neutron_server_tls_proxy container is not persisted to the host

    Before:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    [root@controller-0 ~]#

    After:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    total 136
    -rw-r--r--. 1 root root 400 Nov 15 17:43 error_log
    -rw-r--r--. 1 root root 72784 Nov 15 17:53 neutron-api-proxy_access_ssl.log
    -rw-r--r--. 1 root root 2748 Nov 15 17:43 neutron-api-proxy_error_ssl.log
    [root@controller-0 ~]#

    We backport only the logging bit because this needs to be fixed all the
    way to queens which did not have the ovn-tls work from Kamil.
    (I should have split this neutron api TLS fixes in two commits in the
    first place, but it's too late for that)

    Change-Id: Id9b9a15d40467507bdeace56be55b63a6298b560

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/697334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/697334
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0aaae842beee2468fb726aa92faa8b53d7266a9c
Submitter: Zuul
Branch: stable/rocky

commit 0aaae842beee2468fb726aa92faa8b53d7266a9c
Author: Michele Baldessari <email address hidden>
Date: Wed Dec 4 18:51:43 2019 +0100

    Fix neutron api TLS logging

    This is a partial cherry pick (logging part) of
    Ia1c22b8981fb735a052707516f73313c4e5ca93c
    Closes-Bug: #1852779

    Namely it is the second part of the commit only:
    B) The logging in the neutron_server_tls_proxy container is not persisted to the host

    Before:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    [root@controller-0 ~]#

    After:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    total 136
    -rw-r--r--. 1 root root 400 Nov 15 17:43 error_log
    -rw-r--r--. 1 root root 72784 Nov 15 17:53 neutron-api-proxy_access_ssl.log
    -rw-r--r--. 1 root root 2748 Nov 15 17:43 neutron-api-proxy_error_ssl.log
    [root@controller-0 ~]#

    We backport only the logging bit because this needs to be fixed all the
    way to queens which did not have the ovn-tls work from Kamil.
    (I should have split this neutron api TLS fixes in two commits in the
    first place, but it's too late for that)

    (cherry-picked from 0aecbc772384ae4623fe2013c1698825f268e16b)
    Change-Id: Id9b9a15d40467507bdeace56be55b63a6298b560

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/699356

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/699356
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3c1f10d331444875b3a9bbfa5c052e3ecb124d2d
Submitter: Zuul
Branch: stable/queens

commit 3c1f10d331444875b3a9bbfa5c052e3ecb124d2d
Author: Michele Baldessari <email address hidden>
Date: Wed Dec 4 18:51:43 2019 +0100

    Fix neutron api TLS logging

    This is a partial cherry pick (logging part) of
    Ia1c22b8981fb735a052707516f73313c4e5ca93c
    Closes-Bug: #1852779

    Namely it is the second part of the commit only:
    B) The logging in the neutron_server_tls_proxy container is not persisted to the host

    Before:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    [root@controller-0 ~]#

    After:
    [root@controller-0 ~]# ls -l /var/log/containers/httpd/neutron-api/
    total 136
    -rw-r--r--. 1 root root 400 Nov 15 17:43 error_log
    -rw-r--r--. 1 root root 72784 Nov 15 17:53 neutron-api-proxy_access_ssl.log
    -rw-r--r--. 1 root root 2748 Nov 15 17:43 neutron-api-proxy_error_ssl.log
    [root@controller-0 ~]#

    We backport only the logging bit because this needs to be fixed all the
    way to queens which did not have the ovn-tls work from Kamil.
    (I should have split this neutron api TLS fixes in two commits in the
    first place, but it's too late for that)

    (cherry-picked from 0aecbc772384ae4623fe2013c1698825f268e16b)
    Change-Id: Id9b9a15d40467507bdeace56be55b63a6298b560
    (cherry picked from commit 0aaae842beee2468fb726aa92faa8b53d7266a9c)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.3.1

This issue was fixed in the openstack/tripleo-heat-templates 11.3.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates rocky-eol

This issue was fixed in the openstack/tripleo-heat-templates rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates queens-eol

This issue was fixed in the openstack/tripleo-heat-templates queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates stein-eol

This issue was fixed in the openstack/tripleo-heat-templates stein-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.