Ubuntu
nova package

ceph rbd username and secret should be configured in nova-compute, not passed from nova-volume/cinder

Bug #1065883 reported by James Page on 2012-10-12

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Cinder	Invalid	Undecided	Unassigned
OpenStack Compute (nova)	Fix Released	Undecided	James Page	OpenStack Compute (nova) 2013.1 "grizzly"
cinder (Ubuntu)	Invalid	Undecided	Unassigned
nova (Ubuntu)	Fix Released	Undecided	Unassigned

Bug Description

I'm testing using ceph RADOS block devices to back nova volumes; however I've hit an issue which limits its usefulness in environments where cephx authentication is required.

Configuration is directly taken from http://ceph.com/docs/master/rbd/rbd-openstack/#configuring-cinder-nova-volume. Note that nova-volume and nova-compute are running on different hosts.

The problem is as follows:

The rbd_user and rbd_secret_uuid must be configured in nova-volume to ensure that when the nova-compute nodes attach volumes to instances, they will use the libvirt stored secret.

However, the libvirt secret UUID when created on each of the compute nodes is going to be different; and nova-compute will try to attach using the secret provided from nova-volume - for which it has no knowledge.

I also want to configure nova-compute with a different username to nova-volume/cinder to provide more granular access control to ceph.

The user and secret_uuid should be configured in nova-compute; not provided by nova-volume.

I've worked around this using this patch/hack:

=== modified file 'nova/virt/libvirt/volume.py'
--- nova/virt/libvirt/volume.py 2012-08-27 15:37:18 +0000
+++ nova/virt/libvirt/volume.py 2012-10-12 08:37:38 +0000
@@ -88,9 +88,11 @@
         conf.serial = connection_info.get('serial')
         netdisk_properties = connection_info['data']
         if netdisk_properties.get('auth_enabled'):
- conf.auth_username = netdisk_properties['auth_username']
+ conf.auth_username = FLAGS.rbd_user or \
+ netdisk_properties['auth_username']
             conf.auth_secret_type = netdisk_properties['secret_type']
- conf.auth_secret_uuid = netdisk_properties['secret_uuid']
+ conf.auth_secret_uuid = FLAGS.rbd_secret_uuid or \
+ netdisk_properties['secret_uuid']
         return conf

Which basically allows me to override the auth_username and auth_secret_uuid through the nova-compute configuration file.

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: nova-compute (not installed)
ProcVersionSignature: Ubuntu 3.5.0-17.27-generic 3.5.5
Uname: Linux 3.5.0-17-generic x86_64
ApportVersion: 2.6.1-0ubuntu2
Architecture: amd64
Date: Fri Oct 12 09:38:32 2012
SourcePackage: nova
UpgradeStatus: Upgraded to quantal on 2012-06-11 (122 days ago)

See original description

Tags:

Related branches

lp://staging/~openstack-ubuntu-testing/nova/quantal-folsom-proposed

lp://staging/ubuntu/quantal-proposed/nova

lp://staging/~openstack-ubuntu-testing/nova/precise-folsom-proposed

lp://staging/~openstack-ubuntu-testing/nova/quantal-folsom

lp://staging/~openstack-ubuntu-testing/nova/raring-grizzly

lp://staging/~openstack-ubuntu-testing/nova/precise-folsom

Revision history for this message

James Page (james-page) wrote on 2012-10-12:

ProcEnviron.txt Edit (313 bytes, text/plain; charset="utf-8")

James Page (james-page) on 2012-10-12

description:

updated

Revision history for this message

James Page (james-page) wrote on 2012-10-12:

rbd-security.patch Edit (1.4 KiB, text/plain)

Revised patch which ensures sheepdog handling does not get interfered with

Ubuntu Foundations Team Bug Bot (crichton) on 2012-10-12

tags:

added: patch

Chuck Short (zulcss) on 2012-10-12

Changed in nova (Ubuntu):
status:	New → Invalid
status:	Invalid → Confirmed

Revision history for this message

Josh Durgin (jdurgin) wrote on 2012-10-12:

You can actually specify the uuid for the secret when you add it to libvirt, so it can be the same on all compute hosts.

i.e.

<secret ephemeral='no' private='no'>
  <usage type='ceph'>
    <name>client.volumes secret</name>
  </usage>
  <uuid>a060c8a3-d905-45ec-84a6-0b5d7e25c5cb</uuid>
</secret>

Libvirt only generates a random uuid if you don't specify one. I'll update the Ceph docs to clarify this.

Your patch does make sense if you want to control more finely which rados user's your using on the compute nodes. It's easier than running multiple (cinder|nova)-volume processes, but the long term solution probably involves changing the volume driver to use different rados pools and users based on volume_type or some other configuration.

However, with the current rbd volume driver using only a single pool, I'm not sure how much finer-grained the compute node permissions could be compared to the volume service permissions. What do you have in mind?

BTW, sheepdog and nbd don't have auth support through libvirt, so you don't need to check specifically for rbd in your patch.

Revision history for this message

James Page (james-page) wrote on 2012-10-12:

Hi Josh

I was aware that was possible; however I'm deploying openstack automatically and I don't really want to pass the uuid around between nova-volume and nova-compute nodes.

I simply want to provide each of the compute nodes with the cephx key it needs to use and a generated username - and it will just configure its own set of secrets and configure nova appropriately, overriding the config that nova-volume may/will have sent.

My finer grained access control requirement was really around having different keys for volume/cinder and compute - so if I add/remove additional compute farms I can easily manage the keys on a per role basis.

I guess I was just being hyper-cautious with the rbd check in the patch :-)

Revision history for this message

Josh Durgin (jdurgin) wrote on 2012-10-12:

I see, that makes sense now. It'd be good to get this patch upstream for grizzly.

Revision history for this message

James Page (james-page) wrote on 2012-10-12:

I'll work on doing that as my first code contribution to OpenStack!

James Page (james-page) on 2012-10-15

Changed in nova:
assignee:	nobody → James Page (james-page)
status:	New → In Progress

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-10-15:

This bug was fixed in the package nova - 2012.2-0ubuntu5

---------------
nova (2012.2-0ubuntu5) quantal-proposed; urgency=low

  [ Adam Gandelman ]
  * Move management of /var/lib/nova/volumes from nova-common to
    nova-volume. Ensure it has proper permissions. (LP: #1065320)
  * debian/patches/avoid_setuptools_git_dependency.patch: Remove
    setuptools_git from tools/pip-requires to avoid it being automatically
    added to python-nova's runtime dependencies. (LP: #1059907)

  [ Chuck Short ]
  * debian/patches/rbd-security.patch: Support override of ceph rbd
    user and secret in nova-compute. (LP: #1065883)
  * debian/patches/ubuntu/fix-libvirt-firewall-slowdown.patch: Fix
    refreshing of security groups in libvirt not to block on RPC calls.
    (LP: #1062314)
  * debian/patches/ubuntu/fix-ec2-volume-id-mappings.patch: Read deleted
    snapshot and volume id mappings. (LP: #1065785)
-- Chuck Short <email address hidden> Fri, 12 Oct 2012 12:35:01 -0500

Changed in nova (Ubuntu):
status:	Confirmed → Fix Released

James Page (james-page) on 2012-10-15

Changed in cinder (Ubuntu):
status:	New → Invalid

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-10-22: Fix merged to nova (master)

Reviewed: https://review.openstack.org/14458
Committed: http://github.com/openstack/nova/commit/af51b46b1e08b26c07bd32019e54b9c521cb7813
Submitter: Jenkins
Branch: master

commit af51b46b1e08b26c07bd32019e54b9c521cb7813
Author: James Page <email address hidden>
Date: Mon Oct 15 13:21:55 2012 +0100

Allow local rbd user and secret_uuid configuration

    By default, the rbd_user and rbd_secret_uuid are specified in the
    nova-volume/cinder configuration and passed to nova-compute when
    volumes are attached to instances.

    This change allows these values to be specified locally in
    nova-compute which means access control to RADOS devices in ceph
    can be managed independently from nova-volume/cinder with no
    requirement for consistent uuid's for libvirt secrets.

Fixes bug 1065883.

Change-Id: I9f07d040ae267bfbe8f794a5d22d327106314cc6