s3 backend fails with invalid certificate when using s3 compatible storage

Bug #2030825 reported by Antony Messerli
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
glance_store
Fix Committed
Undecided
Cyril Roelandt

Bug Description

When using the Glance s3 backend, if you are using an s3 compatible store, image operations fail with:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129).

The current implementation uses boto3 and assumes you are only using Amazon's implementation as there are not currently any settings for overriding the CA. In my case, we are using an s3 compatible on-prem device which has internal corporate certs. If I override using an environment variable of AWS_CA_BUNDLE to my CA bundle, the s3 backend then works great.

Can we see about adding an option to the configuration file for the s3_backend so that we can specify the location of a CA bundle so that the default CA can be overridden? It appears a few of the other options have this functionality already, so we would need to add the support for boto3.

This was tested in Antelope and validated to work once the environment variable was added.

Revision history for this message
Cyril Roelandt (cyril-roelandt) wrote :

> It appears a few of the other options have this functionality already, so we would need to add the support for boto3.

What options are you talking about? 's3_store_host' for instance?

What do you mean by "adding support for boto3"? Do you think this feature would require changes to boto3 itself?

I think we could add a "verify=path/to/cert/bundle.pem" (see https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client) argument to the "session.client()" call here https://github.com/openstack/glance_store/blob/0c60291637d1c941dcd8d2e022acb22ba0bed440/glance_store/_drivers/s3.py#L503 .

Revision history for this message
Antony Messerli (antonym) wrote :

No I don’t think we’d need to make changes to boto itself. I think we probably just need to add what you suggested so that we have an option to overide the path to the cert bundle.

I was testing adding support to Kolla for s3 backend and it didn’t seem like anything worked other than setting the ENV var during the container build process so figured the best place to add support for overriding the cert was upstream as a glance configuration.

Revision history for this message
Cyril Roelandt (cyril-roelandt) wrote :

So I put together https://review.opendev.org/c/openstack/glance_store/+/893980 that we may end up discussing during the next PTG if this solves your issue. Could you take a look at it and try it?

Revision history for this message
Alejandro Garcia (agarciaws) wrote :

We recently came across this very same thing (also while trying to integrate a Kolla-based OpenStack with our on-prem S3 Ceph-based). We can confirm the linked patch works and allowed us to use our own CA certificates.

Cyril, could you update the patch in order to revive it? We can also volunteer to do it, as long as someone else reviews it.

Revision history for this message
Cyril Roelandt (cyril-roelandt) wrote :

Hello,

I rebased the patch, let's see if the CI passes. Then it's just a matter of reviewing it :)

Thanks for testing the patch, this helps a lot!

Revision history for this message
Alejandro Garcia (agarciaws) wrote :

Hi Cyril,

We've just tested this again with the rebased patch and everything is working properly. I've already given it my +1.

Thanx to you for fixing this!

affects: glance → glance-store
Changed in glance-store:
assignee: nobody → Cyril Roelandt (cyril-roelandt)
status: New → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance_store (stable/2023.1)

Change abandoned by "Dr. Jens Harbott <email address hidden>" on branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/glance_store/+/935503
Reason: stable/2023.1 branch of this repo is about to be deleted.
To be able to do that, all open patches need to be abandoned.
Please cherry pick the patch to unmaintained/2023.1 if you want to
further work on this patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 4.9.0

This issue was fixed in the openstack/glance_store 4.9.0 Epoxy release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 4.8.2

This issue was fixed in the openstack/glance_store 4.8.2 Dalmatian release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.