Cannot connect Glance because ssl_ca is ignored when enabled-services=volume

Bug #1967302 reported by Nobuto Murata
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Cinder Charm
Fix Released
Undecided
Nobuto Murata

Bug Description

How to reproduce:
1. deploy OpenStack with a separate cinder-volume service (enabled-services=volume) which is usually required for iSCSI/FC backends
2. enable TLS with an external but private CA using ssl_* charm options (i.e. no Vault as root CA nor intermediate CA)
3. create a volume from an image

$ openstack volume create \
    --size 10 \
    --image auto-sync/ubuntu-focal-20.04-amd64-server-20220322-disk1.img \
    test-volume-from-image

the volume goes into an error:

$ openstack volume list --format yaml
- Attached to: []
  ID: 56d0bb94-a0e2-40db-b7a3-1caacdd967f4
  Name: test-volume-from-image
  Size: 10
  Status: error

[/var/log/cinder/cinder-volume.log]
2022-03-31 13:47:13.895 45153 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://192.168.151.151:9292/v2/images/fad1d9da-7a85-405b-84c4-269970e50330: HTTPSConnectionPool(host='192.168.151.151', port=9292): Max retries exceeded with url: /v2/images/fad1d9da-7a85-405b-84c4-269970e50330 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))

Revision history for this message
Nobuto Murata (nobuto) wrote :

A testbed deployment was done with the following steps:

$ juju deploy ./cinder-volume_bundle.yaml

(the bundle is pretty close to https://github.com/openstack/charm-cinder/blob/stable/21.10/tests/bundles/focal-ussuri-volume-only.yaml but without Vault for cinder-volume)

$ juju run-action vault/leader --wait generate-root-ca

to initiate a root CA with Vault and use it for all API endpoint, but again Vault doesn't have a relation with cinder-volume on purpose to simulate the external but private CA to be imported to the cinder-volume unit.

$ juju config cinder-volume \
    ssl_ca="$(juju run --unit vault/leader -- leader-get root-ca | base64)"

To import the Vault root CA as if it's an external CA to cinder-volume. And the CA is expected to be used to verify other OpenStack API endpoints including Glance.

Revision history for this message
Nobuto Murata (nobuto) wrote :
Nobuto Murata (nobuto)
Changed in charm-cinder:
assignee: nobody → Nobuto Murata (nobuto)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-cinder (master)
Changed in charm-cinder:
status: New → In Progress
Revision history for this message
Nobuto Murata (nobuto) wrote :

Subscribing ~field-medium.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-cinder (master)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/836036
Committed: https://opendev.org/openstack/charm-cinder/commit/a0f44d0f90074495427880e7d03d0c324639a8c4
Submitter: "Zuul (22348)"
Branch: master

commit a0f44d0f90074495427880e7d03d0c324639a8c4
Author: Nobuto Murata <email address hidden>
Date: Fri Apr 1 00:49:03 2022 +0900

    Install ssl_ca for volume service only scenarios

    The CA information is necessary to talk to Keystone or Glance from the
    Cinder volume service. This is a follow-up of the following change where
    the Vault certificate relations is assumed but ssl_ca wasn't addressed
    at that point.
    I69f15c3fd164f7114f5498d100b2832caf93fb00

    Closes-Bug: #1967302
    Change-Id: I4d7b3721fe7dfd6f7cdfd364d8c5bc340d38c00f

Changed in charm-cinder:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-cinder (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-cinder/+/836150

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-cinder (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-cinder/+/836151

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-cinder (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-cinder/+/836152

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-cinder (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-cinder/+/836153

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-cinder (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/836150
Committed: https://opendev.org/openstack/charm-cinder/commit/24fdd506116f5f5061b39391c10bde329a62babf
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 24fdd506116f5f5061b39391c10bde329a62babf
Author: Nobuto Murata <email address hidden>
Date: Fri Apr 1 00:49:03 2022 +0900

    Install ssl_ca for volume service only scenarios

    The CA information is necessary to talk to Keystone or Glance from the
    Cinder volume service. This is a follow-up of the following change where
    the Vault certificate relations is assumed but ssl_ca wasn't addressed
    at that point.
    I69f15c3fd164f7114f5498d100b2832caf93fb00

    Closes-Bug: #1967302
    Change-Id: I4d7b3721fe7dfd6f7cdfd364d8c5bc340d38c00f
    (cherry picked from commit a0f44d0f90074495427880e7d03d0c324639a8c4)

tags: added: in-stable-xena
Changed in charm-cinder:
milestone: none → 22.04
Changed in charm-cinder:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-cinder (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/836152
Committed: https://opendev.org/openstack/charm-cinder/commit/6bcb2bf435d467417292411c1ced812494016a7a
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 6bcb2bf435d467417292411c1ced812494016a7a
Author: Nobuto Murata <email address hidden>
Date: Fri Apr 1 00:49:03 2022 +0900

    Install ssl_ca for volume service only scenarios

    The CA information is necessary to talk to Keystone or Glance from the
    Cinder volume service. This is a follow-up of the following change where
    the Vault certificate relations is assumed but ssl_ca wasn't addressed
    at that point.
    I69f15c3fd164f7114f5498d100b2832caf93fb00

    Closes-Bug: #1967302
    Change-Id: I4d7b3721fe7dfd6f7cdfd364d8c5bc340d38c00f
    (cherry picked from commit a0f44d0f90074495427880e7d03d0c324639a8c4)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-cinder (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/836151
Committed: https://opendev.org/openstack/charm-cinder/commit/c7f110bd037d58476e55ab273da43a3ab6fa9b4c
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit c7f110bd037d58476e55ab273da43a3ab6fa9b4c
Author: Nobuto Murata <email address hidden>
Date: Fri Apr 1 00:49:03 2022 +0900

    Install ssl_ca for volume service only scenarios

    The CA information is necessary to talk to Keystone or Glance from the
    Cinder volume service. This is a follow-up of the following change where
    the Vault certificate relations is assumed but ssl_ca wasn't addressed
    at that point.
    I69f15c3fd164f7114f5498d100b2832caf93fb00

    Closes-Bug: #1967302
    Change-Id: I4d7b3721fe7dfd6f7cdfd364d8c5bc340d38c00f
    (cherry picked from commit a0f44d0f90074495427880e7d03d0c324639a8c4)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-cinder (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/836153
Committed: https://opendev.org/openstack/charm-cinder/commit/d9b134bb568c81f94ea93224337fd1fc27d7a270
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit d9b134bb568c81f94ea93224337fd1fc27d7a270
Author: Nobuto Murata <email address hidden>
Date: Fri Apr 1 00:49:03 2022 +0900

    Install ssl_ca for volume service only scenarios

    The CA information is necessary to talk to Keystone or Glance from the
    Cinder volume service. This is a follow-up of the following change where
    the Vault certificate relations is assumed but ssl_ca wasn't addressed
    at that point.
    I69f15c3fd164f7114f5498d100b2832caf93fb00

    Closes-Bug: #1967302
    Change-Id: I4d7b3721fe7dfd6f7cdfd364d8c5bc340d38c00f
    (cherry picked from commit a0f44d0f90074495427880e7d03d0c324639a8c4)

tags: added: in-stable-ussuri
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.