Incorrect configuration in apache2 for internal service port

Bug #2022980 reported by Felipe Reyes
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Charm Helpers
Fix Committed
Undecided
Edward Hope-Morley
OpenStack Ironic API Charm
Fix Committed
Undecided
Unassigned
2023.1
New
Undecided
Unassigned
OpenStack Octavia Charm
Fix Committed
Undecided
Unassigned
2023.1
Fix Committed
Undecided
Unassigned
charms.openstack
Invalid
Undecided
Unassigned

Bug Description

An ironic based environment is misconfigured in the apache configuration, the ssl termination endpoint should be passing the requests to http://localhost:6365/ although it's passing it to http://localhost:6375/, the symptoms seen are:

$ openstack baremetal conductor list
SSL exception connecting to https://10.5.250.100:6385/v1/conductors: HTTPSConnectionPool(host='10.5.250.100', port=6385): Max retries exceeded with url: /v1/conductors (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))

This behavior is currentl reproduced in a lab environment, although this same issue is being seen at the gate[0]

Here is the list of relevant open ports:

```
root@juju-d974dd-zaza-cf05be3e3598-10:/var/log/juju# ss -tpln | grep -P '(6385|6375|6365)'
LISTEN 0 4096 0.0.0.0:6385 0.0.0.0:* users:(("haproxy",pid=136051,fd=7))
LISTEN 0 128 0.0.0.0:6365 0.0.0.0:* users:(("ironic-api",pid=132954,fd=8),("ironic-api",pid=132954,fd=7),("ironic-api",pid=132579,fd=7))
LISTEN 0 4096 *:6385 *:* users:(("haproxy",pid=136051,fd=8))
LISTEN 0 511 *:6375 *:* users:(("apache2",pid=235171,fd=8),("apache2",pid=235170,fd=8),("apache2",pid=235169,fd=8))
```

Here it's the content of /etc/apache2/sites-enabled/openstack_https_frontend.conf:

```
Listen 6375
<VirtualHost 10.5.3.161:6375>
    ServerName 10.5.250.100
    SSLEngine on

    # This section is based on Mozilla's recommendation
    # as the "intermediate" profile as of July 7th, 2020.
    # https://wiki.mozilla.org/Security/Server_Side_TLS
    SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
    SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    SSLHonorCipherOrder off

    SSLCertificateFile /etc/apache2/ssl/ironic/cert_10.5.250.100
    SSLCertificateKeyFile /etc/apache2/ssl/ironic/key_10.5.250.100
    ProxyPass / http://localhost:6375/
    ProxyPassReverse / http://localhost:6375/
    ProxyPreserveHost on
    RequestHeader set X-Forwarded-Proto "https"
    KeepAliveTimeout 75
    MaxKeepAliveRequests 1000
</VirtualHost>
<Proxy *>
    Order deny,allow
    Allow from all
</Proxy>
<Location />
    Order allow,deny
    Allow from all
</Location>
```

Error in the gate:

```

2023-06-01 21:25:46.511409 | focal-medium | 2023-06-01 21:25:46 [INFO] raise exceptions.SSLError(msg)
2023-06-01 21:25:46.511437 | focal-medium | 2023-06-01 21:25:46 [INFO] keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://172.16.220.230:6385/v1/conductors: HTTPSConnectionPool(host='172.16.220.230', port=6385): Max retries exceeded with url: /v1/conductors (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))
```

[0] https://openstack-ci-reports.ubuntu.com/artifacts/80d/884252/8/check/jammy-zed/80dbfa2/

Revision history for this message
Edward Hope-Morley (hopem) wrote :

same issue with octavia on zed/stable:

# egrep "ProxyPass|Listen" /etc/apache2/sites-enabled/openstack_https_frontend.conf
Listen 9866
    ProxyPass / http://localhost:9866/
    ProxyPassReverse / http://localhost:9866/

i.e. it is proxying to itself. Changing those to 9865 fixes it.

Revision history for this message
Edward Hope-Morley (hopem) wrote :

This is the culprit - https://github.com/juju/charm-helpers/commit/6064a34627882d1c8acf74644c48d05db67ee3b4

If i remove that patch it works again.

Revision history for this message
Edward Hope-Morley (hopem) wrote :

The problem is that the above patch expects the following key to exist on the certificates relation:

octavia_1.processed_requests

But e.g. the octavia charm only has:

$ juju run --format json -u octavia/1 -- relation-get -r certificates:49 - vault/0| jq -r '.[].Stdout'| egrep "^\w+"
ca: |-
chain: |-
client.cert: |-
client.key: |-
egress-subnets: 10.5.2.221/32
ingress-address: 10.5.2.221
octavia_0.server.cert: |-
octavia_0.server.key: |-
octavia_1.server.cert: |-
octavia_1.server.key: |-
octavia_2.server.cert: |-
octavia_2.server.key: |-
private-address: 10.5.2.221

so it thinks it is not an https endpoint.

Revision history for this message
Felipe Reyes (freyes) wrote :

> i.e. it is proxying to itself. Changing those to 9865 fixes it.

yep, that's the behavior I observed. I'm rebuilding without the patch you pointed out, sadly I already destroyed my environment, so I can't check if the process_requests flag is set in my environment.

Revision history for this message
Edward Hope-Morley (hopem) wrote :
Changed in charm-helpers:
status: New → In Progress
assignee: nobody → Edward Hope-Morley (hopem)
Felipe Reyes (freyes)
Changed in charms.openstack:
status: New → Invalid
Felipe Reyes (freyes)
Changed in charm-helpers:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (master)
Changed in charm-ironic-api:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ironic-api (master)

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/885518
Committed: https://opendev.org/openstack/charm-ironic-api/commit/6d7d772c2a6a0a936c87ad0ce4ca7ab8422d2561
Submitter: "Zuul (22348)"
Branch: master

commit 6d7d772c2a6a0a936c87ad0ce4ca7ab8422d2561
Author: Felipe Reyes <email address hidden>
Date: Wed Jun 7 11:43:23 2023 -0400

    Rebuild charm

    Closes-Bug: #2022980
    Change-Id: I3a69ad9f6725fef0a8ace253e793ea1aeea20bfa

Changed in charm-ironic-api:
status: In Progress → Fix Committed
Revision history for this message
Edward Hope-Morley (hopem) wrote :

now that https://review.opendev.org/c/openstack/charm-octavia/+/883868 is landed it will trigger a rebuild so we can mark octavia as done.

Changed in charm-octavia:
status: New → Fix Committed
Revision history for this message
Jadon Naas (jadonn) wrote :

There was a backport to 2023.1 that was merged according to this review: https://review.opendev.org/c/openstack/charm-octavia/+/886091. I updated the status on this bug for charm-octavia for 2023.1

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.