Concurrency Update issue in v2

Bug #1371728 reported by Alexander Tivelkov
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Glance
In Progress
High
Darja Shakhray

Bug Description

UpdateImage call in v2 ( PATCH /v2/images/%imageid%) fetches the whole image object from database (https://github.com/openstack/glance/blob/master/glance/api/v2/images.py#L119), modifies some of its properties with appropriate matching methods (_do_replace, _do_add etc) and then saves the object in ImageRepo

The imageRepo takes the object, renders it to dictionary (https://github.com/openstack/glance/blob/master/glance/db/__init__.py#L169) and then passes this dictionary to db_api.image_update(), which eventually executes UPDATE query and replaces all the fields in database with these values.

This is not transactional, and other operations may interfere between image fetching and update query.

For example, imagine we have an image with name "test" and min_ram "10".
If we concurrently execute two image-update requests:
  'image-update --name test2'
and
  'image-update --min-ram 100'

A race condition occurs. The result image is undetermined: it may be either {'name': 'test', 'min_ram': 100} or {'name': 'test2', 'min_ram': 10} or {'name': 'test2', 'min_ram': 100'} - depending on which request fetches the object first.
It may be had to reproduce this in normal circumstances, but once the database load becomes higher, it may show up.
To reproduce it for sure one may add a event let.sleep() command after line 119 of glance/api/v2/images.py, to increase the timespan between image fetching and image update.

Tags: kilo-1
description: updated
Changed in glance:
assignee: nobody → Alexander Tivelkov (ativelkov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/122814

Changed in glance:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to glance (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/123722

Zhi Yan Liu (lzy-dev)
Changed in glance:
importance: Undecided → High
Thierry Carrez (ttx)
tags: added: juno-rc-potential
Thierry Carrez (ttx)
tags: added: juno-backport-potential
removed: juno-rc-potential
Revision history for this message
Erno Kuvaja (jokke) wrote :

Had quick chat with Thierry and we came to conclusion the solution for this is not worth of proceeding backport to stable. Thus removed the tag.

tags: removed: juno-backport-potential
tags: added: kilo-1
Changed in glance:
milestone: none → kilo-2
Thierry Carrez (ttx)
Changed in glance:
milestone: kilo-2 → kilo-3
Thierry Carrez (ttx)
Changed in glance:
milestone: kilo-3 → kilo-rc1
tags: added: kilo-rc-potential
Changed in glance:
milestone: kilo-rc1 → none
Thierry Carrez (ttx)
tags: removed: kilo-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance (master)

Change abandoned by Alexander Tivelkov (<email address hidden>) on branch: master
Review: https://review.openstack.org/123722

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Glance Bot (<email address hidden>) on branch: master
Review: https://review.openstack.org/122814

Changed in glance:
assignee: Alexander Tivelkov (ativelkov) → Darja Shakhray (dshakhray)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.