Concurrently update server's metadata are handled badly
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
In Progress
|
Medium
|
Unassigned |
Bug Description
Currently, we have two APIs to update server's metadata:
1. update: only update/add the key=value user provided in this call
2. update_all: replace all previous added key=value pair with the key=value pair user provided in this call
they are using the same _update_
http://
http://
Then it will be handled in the below work flow:
I. get the server object
http://
II. handled in compute/api.py update_
http://
the difference of update_all and update is handled here:
i) if we are using update all, the target meatadata will be set as the metadata passed in, say new_meta
ii) if we are using update, the target meatadata will be set to old_meta + new_meta
III. then we just set instance.metadata to the target metadata, and call instance.save() to do the job.
IV. it is finally handled in DB:
http://
here we compared the metadata read from DB and the target metadata we passed in:
i) if we call update_all, the target metadata only contains new_meta, so key=value pairs that are in old_meta but
not in new_meta will be deleted.
ii) if we call update, the target metadata will contains new_meta + old_meta so old_meta will not be deleted.
The above mentioned process worked pretty well for general uses, but when we come to concurrently usage, problem may
occour:
For example, we have two concurrent meta_data update calls, say meta_A and meta_B, they are called at the same time
since the target meta is generated at the API level(in previous step I and II), it the two calls came in at the same
time, the instance.metadata got in these two calls will be the same (old_meta), here comes the problem, the target
meta for A call will be old_meta + A, target meta for B call will be old_meta + B, and they will then go the rest
of the process;
When it comes to the DB layer, step IV, as we only have one DB so it is first time come first serve style, lets say
call A has successfully handled, the metadata in DB is now old_meta + A, then we will handle call B, as the target
meta is old_meta + B hence A will be removed.
Changed in nova: | |
assignee: | nobody → Zhenyu Zheng (zhengzhenyu) |
Changed in nova: | |
assignee: | Yikun Jiang (yikunkero) → nobody |
I'm not sure we can solve this easily without introducing optimistic concurrency control to metadata REST API, e.g. something similar to e-tags, so that the caller has to provide an e-tag on update and will get 409 Conflict, if a concurrent request has been completed first.