OpenStack Compute (nova)

Live migration failed in openstack on xenserver using microversion 2.34

  1. Series newton
  2. Bug #1709236
Bug #1709236 reported by Naichuan Sun
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Undecided
Unassigned
Newton
Confirmed
Undecided
Unassigned
Ocata
Confirmed
Undecided
Unassigned

Bug Description

XenServer live migration is broken with the latest NOVA API. At least it works well when using Nova API 2.27.

See the following trace where it reports error with "Action for request_id req-a06c9561-0458-43c6-b767-08bf67e38b07 on instance 8a8ed9ad-2fb8-46d4-bee2-9d947e2d3e58 not found"

ul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server [None req-a06c9561-0458-43c6-b767-08bf67e38b07 admin admin] Exception during message handling: InstanceActionNotFound: Action for request_id req-a06c9561-0458-43c6-b767-08bf67e38b07 on instance 8a8ed9ad-2fb8-46d4-bee2-9d947e2d3e58 not found
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 863, in decorated_function
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server with EventReporter(context, event_name, instance_uuid):
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 834, in __enter__
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server self.context, uuid, self.event_name, want_result=False)
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server result = fn(cls, context, *args, **kwargs)
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/objects/instance_action.py", line 169, in event_start
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server db_event = db.action_event_start(context, values)
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/api.py", line 1958, in action_event_start
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server return IMPL.action_event_start(context, values)
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 250, in wrapped
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server return f(context, *args, **kwargs)
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 6155, in action_event_start
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server instance_uuid=values['instance_uuid'])
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server InstanceActionNotFound: Action for request_id req-a06c9561-0458-43c6-b767-08bf67e38b07 on instance 8a8ed9ad-2fb8-46d4-bee2-9d947e2d3e58 not found
Jul 27 01:57:12 DevStackOSDomU nova-conductor[2134]: ERROR oslo_messaging.rpc.server

See original description
Revision history for this message
Naichuan Sun (naichuans) wrote :

stack@DevStackOSDomU:~$ nova instance-action-list 04f7d20b-7c47-4c81-b7bc-e601f6db1870
+----------------+------------------------------------------+---------+----------------------------+
| Action | Request_ID | Message | Start_Time |
 +----------------+------------------------------------------+---------+----------------------------+
| create | req-31b8da65-2e44-4475-a4b0-bb3655b0305a | - | 2017-07-27T02:10:21.000000 |
| live-migration | req-54635c3c-dc43-4244-a0d6-c1da0d68c19c | - | 2017-07-27T02:13:22.000000 |
 +----------------+------------------------------------------+---------+----------------------------+

But we can see the nova CLI is using different API: DEBUG (connectionpool:395) http://10.71.217.6:80 "GET /compute/v2.1/servers/04f7d20b-7c47-4c81-b7bc-e601f6db1870/os-instance-actions HTTP/1.1" 200 619

affects: linuxmint → nova
Revision history for this message
Naichuan Sun (naichuans) wrote :

Seems this issue doesn't exist with nova API2.0.
So a workaround to verify other change is to force it to use API2.0: export OS_COMPUTE_API_VERSION=2.0

Revision history for this message
Jianghua Wang (wjh-fresh) wrote :

I did a test with V2.27 and the latest version V2.53.
The live-migration works well with V2.27 but failed with V2.53 with the same error reported by Naichuan:

Aug 08 05:23:13 DevStackOSDomU nova-conductor[4417]: ERROR oslo_messaging.rpc.server InstanceActionNotFound: Action for request_id req-9dd0d0e2-4b92-46d3-ad95-2aa89eeebde5 on instance 81a0ef82-8127-4b51-9f8d-4bfe7d429149 not found

Revision history for this message
Naichuan Sun (naichuans) wrote :

According to Jianghua's testing, the live migration would be failed in current CI test. The default nova API version is 2.1. The reason is same with the logs in the description.
He also tested other nova APIs. version 2.1 and 2.27 failed while version 2.0 and version 2.53 work well.

Revision history for this message
Naichuan Sun (naichuans) wrote :

Sorry, the nova API version 2.0 and 2.27 work well, 2.1 and 2.53 would fail.

Jianghua Wang (wjh-fresh)
description: updated
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Looks like the compute service provides the exception, but you only give us the conductor one. Could you please provide the nova-compute.log that returns this exception so we can verify which decorated RPC method returns this ?

Also, please note we changed in v2.34 how live-migration pre-flight checks were done. See http://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/async-live-migration-rest-check.html for the detailed reasoning behind that and https://review.openstack.org/#/c/314932/18/releasenotes/notes/async-live-migration-rest-check-675ec309a9ccc28e.yaml for the according release note.

For some reason, looks like the instance action record wasn't correctly persisted, so when trying to get it for adding the start event, it fails...

Changed in nova:
status: New → Incomplete
Sylvain Bauza (sylvain-bauza)
tags: added: live-migration
Revision history for this message
Naichuan Sun (naichuans) wrote :

Hi, Sylvain,
The whole log for nova is in the attachment.
We checked the Nova API versions, 2.33 works well, 2.34 failed. So it should be related to the modification you mentioned.
Thank you very much.

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Putting back the new state since we now have the logs and that issue could be sensitive.

Changed in nova:
status: Incomplete → New
Matt Riedemann (mriedem)
Changed in nova:
status: New → Confirmed
Sylvain Bauza (sylvain-bauza)
tags: added: xenserver
tags: added: openstack-version.newton
Matt Riedemann (mriedem)
summary: - Live migration failed in openstack on xenserver
+ Live migration failed in openstack on xenserver using microversion 2.34
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.