cinder "forgets" about ceph following upgrade from ~15.04 to 17.11

Bug #1753611 reported by Paul Collins
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Cinder Charm
Fix Released
High
Craige McWhirter

Bug Description

Following an upgrade of a ~15.04 environment to 17.11, cinder (originally r91, revision-id: <email address hidden>) stopped working. I could no longer create, delete, detach, or attach volumes. Although cinder is related to ceph and block-devices is set to "None", following the upgrade, the service was configured to use a non-existent "LVM" backend definition:

root@juju-pjdc-ps45-1711-test-machine-7:~# grep enabled_backends /etc/cinder/cinder.conf
enabled_backends = LVM
root@juju-pjdc-ps45-1711-test-machine-7:~# grep '^\[' /etc/cinder/cinder.conf
[DEFAULT]
[keystone_authtoken]
[database]
root@juju-pjdc-ps45-1711-test-machine-7:~# _

The Ceph configuration was still present in the "DEFAULT" section, and simply commenting out "enabled_backends = LVM" and restarting the cinder services made it work again, but of course Juju may revert this at any time.

At first I thought I was required to deploy cinder-ceph following the upgrade, and although this restored cinder to life, as far as I can tell from reading the charm release notes and consulting https://jujucharms.com/cinder/ the simple case of a single Ceph cluster directly related to cinder seems like it is intended to remain supported.

If attempting to reproduce, run the following (adjusted as needed to match the cinder config) before upgrading cinder to 17.11. The version of the charmhelpers code as used in 15.04 did not set request-id as expected and required by the later versions, and so the 17.11 version of the ceph charm will not process the request.

juju run --unit cinder/0 'relation-set -r $(relation-ids ceph) broker_req="{\"api-version\": 1, \"request-id\": \"'$(uuid -v1)'\", \"ops\": [{\"replicas\": 3, \"name\": \"cinder\", \"op\": \"create-pool\"}]}"'

I also tried breaking and remaking the cinder<->ceph relation, to no effect.

Paul Collins (pjdc)
tags: added: canonical-is-ps45-1711-upgrade
Revision history for this message
Paul Collins (pjdc) wrote :

This also affects a fresh icehouse deploy with the 17.11 charms.

Here's a fix (I forgot, yet again, that changes go through Gerrit. I'll look into that): https://github.com/openstack/charm-cinder/pull/2

Xav Paice (xavpaice)
tags: added: canonical-bootstack
Revision history for this message
Paul Collins (pjdc) wrote :

To be clear, I tested the patch above, and also it is now live on our newly-upgraded-to-17.11 production cloud.

Revision history for this message
Craige McWhirter (craige) wrote :

With Paul's permission, I submitted https://review.openstack.org/553115 for review to close this bug.

James Page (james-page)
Changed in charm-cinder:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Craige McWhirter (craige)
status: Triaged → In Progress
James Page (james-page)
Changed in charm-cinder:
milestone: none → 18.05
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-cinder (master)

Reviewed: https://review.openstack.org/553115
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=8369ea875ca51aac4552ec0e84b944db9a17a7e7
Submitter: Zuul
Branch: master

commit 8369ea875ca51aac4552ec0e84b944db9a17a7e7
Author: Craige McWhirter <email address hidden>
Date: Thu Mar 15 11:37:49 2018 +1000

    Only enable the default backend for ocata+

    When deploying on icehouse with Ceph, the RBD backend is
    configured globally, but cinder sets enabled_backends to "LVM",
    which exists (and is wrong to boot).

    This commit moves the check for empty "backends" under the ocata
    check. Per comment #5 on LP:1719742: "The cinder-volume service
    fails to start in ocata+ if enable-backends is not set."

    Closes-Bug: #1753611

    Change-Id: I07a9f9c739064bc93870a1f8f68a7624b7ce3033
    Signed-off-by: Craige McWhirter <email address hidden>

Changed in charm-cinder:
status: In Progress → Fix Committed
David Ames (thedac)
Changed in charm-cinder:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.