rgw trust forwarded https not set

Bug #2021560 reported by Samuel Walladge
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceph RADOS Gateway Charm
Fix Committed
Undecided
Samuel Walladge
Quincy.2
Fix Released
Undecided
Unassigned

Bug Description

For use with encryption, the `rgw trust forwarded https` option must be enabled.

> Requests for server-side encryption must be sent over a secure HTTPS connection to avoid sending secrets in plaintext. If a proxy is used for SSL termination, rgw trust forwarded https must be enabled before forwarded requests will be trusted as secure.

https://docs.ceph.com/en/latest/radosgw/encryption/

This option is not currently set by the charm. It should be set, so that SSE-C encryption can work out of the box.

It should be always set if the apache2 proxy with ssl is being used. Not sure the security implications of setting it if ssl is not enabled.

---

Note: a current workaround is to run this:

```
juju config ceph-radosgw config-flags='{"global":{"rgw trust forwarded https": true}}'
```

Which configures /etc/ceph/ceph.conf as required with

```
[global]
rgw trust forwarded https = True
```

description: updated
Revision history for this message
Nobuto Murata (nobuto) wrote :

Here is the config reference:

https://docs.ceph.com/en/latest/radosgw/config-ref/?highlight=trust%20forwarded%20https#confval-rgw_trust_forwarded_https
> Enable this option to trust the Forwarded and X-Forwarded-Proto headers sent by the proxy when determining whether the connection is secure. This is required for some features, such as server side encryption. (Never enable this setting if you do not have a trusted proxy in front of radosgw, or else malicious users will be able to set these headers in any request.)

And the charm has the following for Apache2 as a reverse proxy and TLS termination:
https://github.com/openstack/charm-ceph-radosgw/blob/4484b0f0ed0c3b32947cfaa96025dbcc489c4f14/templates/openstack_https_frontend.conf#L24
> RequestHeader set X-Forwarded-Proto "https"

So I think we can enable the config when TLS is configured by the charm.

Revision history for this message
Samuel Walladge (swalladge) wrote :

For reference, this is the output from the minio cli when the trust forwarded https is not enabled:

```
$ mcli --insecure cp mcli_20230526233154.0.0_amd64.deb testbed/mybucket/encrypted-mcli2.deb --encrypt-key "testbed/mybucket/=JkjkEoXp0HC3YYVFm7IKaOoG/18ZvI8bQ/y+ioPNx2s="
mcli: <ERROR> Failed to copy `mcli_20230526233154.0.0_amd64.deb`. Error response code InvalidRequest.

# and debug output for the last request
mcli: <DEBUG> PUT /mybucket/encrypted-mcli2.deb HTTP/1.1
Host: 192.168.151.118
User-Agent: MinIO (linux; amd64) minio-go/v7.0.55 mcli/RELEASE.2023-05-26T23-31-54Z
Content-Length: 9862956
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20230530/default/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-server-side-encryption-customer-algorithm;x-amz-server-side-encryption-customer-key;x-amz-server-side-encryption-customer-key-md5, Signature=**REDACTED**
Content-Type: application/x-debian-package
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20230530T065040Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: JkjkEoXp0HC3YYVFm7IKaOoG/18ZvI8bQ/y+ioPNx2s=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: DqFKVxJ0Q63IcSfbi3fKnw==
Accept-Encoding: gzip

mcli: <DEBUG> HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 220
Accept-Ranges: bytes
Content-Type: application/xml
Date: Tue, 30 May 2023 06:50:40 GMT
Server: Apache/2.4.52 (Ubuntu)
X-Amz-Request-Id: tx00000b40b01d62d21cfb9-0064759cc0-2d9f-default

<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><BucketName>mybucket</BucketName><RequestId>tx00000b40b01d62d21cfb9-0064759cc0-2d9f-default</RequestId><HostId>2d9f-default-default</HostId></Error>mcli: <DEBUG> TLS Certificate found:
mcli: <DEBUG> >> Expires: 2024-05-29 03:59:43 +0000 UTC
mcli: <DEBUG> Response Time: 25.577889ms

```

And when it is, it works fine:

```
$ mcli --insecure cp mcli_20230526233154.0.0_amd64.deb testbed/mybucket/encrypted-mcli2.deb --encrypt-key "testbed/mybucket/=JkjkEoXp0HC3YYVFm7IKaOoG/18ZvI8bQ/y+ioPNx2s="
...0230526233154.0.0_amd64.deb: 9.41 MiB / 9.41 MiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 28.95 MiB/s 0s
```

Revision history for this message
Nobuto Murata (nobuto) wrote :

Subscribing ~field-high.

It's a cosmetic issue indeed and it's easy to workaround by passing config-flag. But I personally think that the charmed radosgw deployment has a deviation from the expected deployment by the upstream radosgw and unnecessarily blocks a documented function of radosgw as Amazon SSE-C.

https://docs.ceph.com/en/quincy/radosgw/encryption/#customer-provided-keys
> no special Ceph configuration is needed to support this encryption mode

Revision history for this message
Samuel Walladge (swalladge) wrote :
Changed in charm-ceph-radosgw:
status: New → In Progress
Changed in charm-ceph-radosgw:
assignee: nobody → Samuel Walladge (swalladge)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceph-radosgw (master)

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/884797
Committed: https://opendev.org/openstack/charm-ceph-radosgw/commit/541ceec4018e311cef1517a62eefa28cd53bc162
Submitter: "Zuul (22348)"
Branch: master

commit 541ceec4018e311cef1517a62eefa28cd53bc162
Author: Samuel Walladge <email address hidden>
Date: Wed May 31 12:23:11 2023 +0930

    Enable rgw trust forwarded https when https proxy

    This option is required for server-side encryption to be allowed
    if radosgw is behind a reverse proxy,
    such as here when certificates are configured and apache2 is running.

    ref. https://docs.ceph.com/en/latest/radosgw/encryption/

    It is safe to always enable when https is configured in the charm,
    because it will be securely behind the reverse proxy in the unit.
    This option must not be enabled when https is not configured in the charm,
    because this would allow clients to spoof headers.

    Closes-Bug: #2021560
    Change-Id: I940f9b2f424a3d98936b5f185bf8f87b71091317

Changed in charm-ceph-radosgw:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceph-radosgw (stable/quincy.2)

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/884827

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceph-radosgw (stable/quincy.2)

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/884827
Committed: https://opendev.org/openstack/charm-ceph-radosgw/commit/13664c11d01976d1b711b8bea184ca2d7a55c10e
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit 13664c11d01976d1b711b8bea184ca2d7a55c10e
Author: Samuel Walladge <email address hidden>
Date: Wed May 31 12:23:11 2023 +0930

    Enable rgw trust forwarded https when https proxy

    This option is required for server-side encryption to be allowed
    if radosgw is behind a reverse proxy,
    such as here when certificates are configured and apache2 is running.

    ref. https://docs.ceph.com/en/latest/radosgw/encryption/

    It is safe to always enable when https is configured in the charm,
    because it will be securely behind the reverse proxy in the unit.
    This option must not be enabled when https is not configured in the charm,
    because this would allow clients to spoof headers.

    Closes-Bug: #2021560
    Change-Id: I940f9b2f424a3d98936b5f185bf8f87b71091317
    (cherry picked from commit 541ceec4018e311cef1517a62eefa28cd53bc162)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.