Fuel for OpenStack

Cannot create volume from glance image without checksum

Bug #1308058 reported by git-harry
112
This bug affects 19 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
Mike Perez
Cinder 2014.2 "juno"
Icehouse
Fix Released
High
Mike Perez
Cinder 2014.1.1
Fuel for OpenStack
Fix Committed
High
Roman Vyalov
Fuel for OpenStack 5.0
OpenStack Compute (nova)
Fix Released
Undecided
Andrew Woodward
OpenStack Compute (nova) 2014.2 "juno"
Icehouse
Fix Released
Undecided
Vish Ishaya
OpenStack Compute (nova) 2014.1.3

Bug Description

It is no longer possible to create a volume from an image that does not have a checksum set.

https://github.com/openstack/cinder/commit/da13c6285bb0aee55cfbc93f55ce2e2b7d6a28f2 - this patch removes the default of None from the getattr call.

If this is intended it would be nice to see something more informative in the logs.

2014-04-15 11:52:26.035 19000 ERROR cinder.api.middleware.fault [req-cf0f7b89-a9c1-4a10-b1ac-ddf415a28f24 c139cd16ac474d2184237ba837a04141 83d5198d5f5a461798c6b843f57540d
f - - -] Caught error: checksum
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault Traceback (most recent call last):
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 75, in __call__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault return req.get_response(self.application)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault application, catch_exc_info=False)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault app_iter = application(self.environ, start_response)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault return resp(environ, start_response)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 615, in __call__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault return self.app(env, start_response)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault return resp(environ, start_response)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault return resp(environ, start_response)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault response = self.app(environ, start_response)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault return resp(environ, start_response)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault return self.func(req, *args, **kwargs)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/openstack/wsgi.py", line 895, in __call__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault content_type, body, accept)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/openstack/wsgi.py", line 943, in _process_stack
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault action_result = self.dispatch(meth, request, action_args)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/openstack/wsgi.py", line 1019, in dispatch
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault return method(req=request, **action_args)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/api/v2/volumes.py", line 346, in create
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault **kwargs)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/volume/api.py", line 189, in create
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault flow_engine.run()
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/taskflow/utils/lock_utils.py", line 54, in wrapper
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault return f(*args, **kwargs)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 96, in run
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault self._run()
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 111, in _ru
n
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault misc.Failure.reraise_if_any(failures.values())
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 649, in reraise_if_any
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault failures[0].reraise()
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 656, in reraise
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault six.reraise(*self._exc_info)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 34, in _e
xecute_task
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault result = task.execute(**arguments)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/volume/flows/api/create_volume.py", line 341, in execute
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault self._check_image_metadata(context, image_id, size)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/volume/flows/api/create_volume.py", line 180, in _check_image_metadata
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault image_meta = self.image_service.show(context, image_id)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/image/glance.py", line 228, in show
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault base_image_meta = self._translate_from_glance(image)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/image/glance.py", line 336, in _translate_from_glance
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault image_meta = _extract_attributes(image)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/cinder/cinder/image/glance.py", line 434, in _extract_attributes
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault output[attr] = getattr(image, attr)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/python-glanceclient/glanceclient/openstack/common/apiclient/base.py", line 462, in __getattr__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault return self.__getattr__(k)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault File "/opt/stack/python-glanceclient/glanceclient/openstack/common/apiclient/base.py", line 464, in __getattr__
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault raise AttributeError(k)
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault AttributeError: checksum
2014-04-15 11:52:26.035 19000 TRACE cinder.api.middleware.fault
2014-04-15 11:52:26.037 19000 INFO cinder.api.middleware.fault [req-cf0f7b89-a9c1-4a10-b1ac-ddf415a28f24 c139cd16ac474d2184237ba837a04141 83d5198d5f5a461798c6b843f57540df - - -] http://162.13.156.43:8776/v2/83d5198d5f5a461798c6b843f57540df/volumes returned with HTTP 500

Mike Perez (thingee)
Changed in cinder:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Mike Perez (thingee) wrote :

Jumped onto this one too fast. How are you ending up with an image without a checksum? Just trying to reproduce this.

Changed in cinder:
status: Triaged → New
importance: High → Undecided
Revision history for this message
git-harry (git-harry) wrote :

By not specifying one when I do glance image-create.

Revision history for this message
Mike Perez (thingee) wrote :

You can create an image with a name and it'll attribute error on that as well.

Changed in cinder:
status: New → Triaged
importance: Undecided → High
Mike Perez (thingee)
Changed in cinder:
assignee: nobody → Mike Perez (thingee)
tags: added: icehouse-backport-potential
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Related fix proposed to cinder (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/88089

Changed in cinder:
status: Triaged → In Progress
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/88090

Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/88089
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=993facb3bf753d23332236dc05b397850bf99144
Submitter: Jenkins
Branch: master

commit 993facb3bf753d23332236dc05b397850bf99144
Author: Mike Perez <email address hidden>
Date: Thu Apr 17 18:46:36 2014 -0700

    Fallback to None on missing Glance image attrs

    It's possible for glance images to be missing attributes like name and
    checksum. We'll just set those to None by default to avoid missing key
    exceptions being raised.

    Closes-Bug: #1308058
    Change-Id: I85c42f8351763da201021a22f5ff0ebd62c6b2db

Changed in cinder:
status: In Progress → Fix Committed
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix proposed to cinder (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/90644

Mike Perez (thingee)
Changed in cinder:
milestone: none → juno-1
Andrew Woodward (xarses)
Changed in nova:
assignee: nobody → Andrew Woodward (xarses)
status: New → In Progress
Dmitry Borodaenko (angdraug)
Changed in fuel:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Fuel OSCI Team (fuel-osci)
milestone: none → 5.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/93649

Revision history for this message
Andrew Woodward (xarses) wrote :

https://review.openstack.org/70518 caused the same issue in nova

Revision history for this message
Andrew Woodward (xarses) wrote :

This was also found in fuel.

@dborodaenko posted in https://bugs.launchpad.net/fuel/+bug/1319046 (now a duplicate)

The problem is with this commit (in Nova and Cinder respectively):
https://review.openstack.org/70518
https://review.openstack.org/84937

As can be seen from cinder debug output, metadata for TestVM image doesn't have 'deleted' attribute:
2014-05-13T12:42:11.397595+00:00 debug: {"status": "active", "name": "TestVM", "tags": [], "checksum": "9d526403713579f446434dd6190f9e9e", "created_at": "2014-05-12T13:54:12Z", "size": 14811136, "murano_image_info": "{\"title\": \"Murano Demo\", \"type\": \"cirros.demo\"}", "updated_at": "2014-05-12T13:54:13Z", "visibility": "public", "self": "/v2/images/43f6b4be-b689-46b2-9dab-412e929d8b6f", "container_format": "bare", "min_disk": 0, "protected": false, "min_ram": 64, "file": "/v2/images/43f6b4be-b689-46b2-9dab-412e929d8b6f/file", "disk_format": "qcow2", "owner": "4d52167746af4dd58a65a0370a6efaa5", "direct_url": "rbd://7361c191-9dda-4646-8798-a226439911b9/images/43f6b4be-b689-46b2-9dab-412e929d8b6f/snap", "id": "43f6b4be-b689-46b2-9dab-412e929d8b6f", "schema": "/v2/schemas/image"}

Which causes AttributeError: deleted

Mike Scherbakov (mihgen)
Changed in fuel:
status: In Progress → Confirmed
Roman Vyalov (r0mikiam)
Changed in fuel:
assignee: Fuel OSCI Team (fuel-osci) → Vitaly Parakhin (vparakhin)
Revision history for this message
Vitaly Parakhin (vparakhin) wrote :

cinder
http://gerrit.vm.mirantis.net:8080/#/c/15570/

Regarding nova - it appears that an appropriate patch already applied to our code - could you please check and confirm?

Changed in fuel:
status: Confirmed → In Progress
Vitaly Parakhin (vparakhin)
Changed in fuel:
assignee: Vitaly Parakhin (vparakhin) → Andrew Woodward (xarses)
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Vitaly, Nova stable/icehouse doesn't have the commit that has introduced this bug in the first place (it's only present in master), so we don't need to patch it. We only need to patch Cinder.

Mike Scherbakov (mihgen)
Changed in fuel:
assignee: Andrew Woodward (xarses) → Roman Vyalov (r0mikiam)
Roman Vyalov (r0mikiam)
Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/93649
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ef3f158a5aa1988be8ab2f295252373f3ee85809
Submitter: Jenkins
Branch: master

commit ef3f158a5aa1988be8ab2f295252373f3ee85809
Author: Andrew Woodward <email address hidden>
Date: Wed May 14 19:22:48 2014 -0700

    Catch missing Glance image attrs with None

    Glance is known to drop values from its response if they don't resolve to true.
     Previously in I67b7dd16a94fe60d873c012f6bd246ab24500d5a the None catch was
     removed as this WILL cause a call to glance for missing value and
     optimizations where made to attempt to pre-cache values that can be missing
     in expected ways. However there are still situations that can't be optimized.

    Notably this can occur with disk_format, container_format, name, deleted,
     and checksum.

    Test adapted from I85c42f8351763da201021a22f5ff0ebd62c6b2db by Mike Perez

    Closes-bug #1308058
    Co-Author: Mike Perez <email address hidden>

    Change-Id: Icddead6f746d4d3ba652858eaae425acce8c177d

Changed in nova:
status: In Progress → Fix Committed
Alan Pevec (apevec)
tags: removed: icehouse-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/icehouse)

Reviewed: https://review.openstack.org/90644
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=1e488339ded20a8bce602202bde7244f1c1b249e
Submitter: Jenkins
Branch: stable/icehouse

commit 1e488339ded20a8bce602202bde7244f1c1b249e
Author: Mike Perez <email address hidden>
Date: Thu Apr 17 18:46:36 2014 -0700

    Fallback to None on missing Glance image attrs

    It's possible for glance images to be missing attributes like name and
    checksum. We'll just set those to None by default to avoid missing key
    exceptions being raised.

    Closes-Bug: #1308058
    Change-Id: I85c42f8351763da201021a22f5ff0ebd62c6b2db
    (cherry picked from commit 993facb3bf753d23332236dc05b397850bf99144)

Thierry Carrez (ttx)
Changed in nova:
milestone: none → juno-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/116425

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/116425
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e1d6e1881b08827d1be95e35ed064364d222659e
Submitter: Jenkins
Branch: stable/icehouse

commit e1d6e1881b08827d1be95e35ed064364d222659e
Author: Andrew Woodward <email address hidden>
Date: Wed May 14 19:22:48 2014 -0700

    Catch missing Glance image attrs with None

    Glance is known to drop values from its response if they don't resolve to true.
     Previously in I67b7dd16a94fe60d873c012f6bd246ab24500d5a the None catch was
     removed as this WILL cause a call to glance for missing value and
     optimizations where made to attempt to pre-cache values that can be missing
     in expected ways. However there are still situations that can't be optimized.

    Notably this can occur with disk_format, container_format, name, deleted,
     and checksum.

    Test adapted from I85c42f8351763da201021a22f5ff0ebd62c6b2db by Mike Perez

    Closes-bug #1308058
    Co-Author: Mike Perez <email address hidden>

    Change-Id: Icddead6f746d4d3ba652858eaae425acce8c177d
    (cherry picked from commit ef3f158a5aa1988be8ab2f295252373f3ee85809)

tags: added: in-stable-icehouse
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-1 → 2014.2
Thierry Carrez (ttx)
Changed in cinder:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.