live block migration of instance with vfat config drive fails
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
High
|
Lee Yarwood | ||
Ocata |
Fix Committed
|
High
|
Matt Riedemann | ||
Pike |
Fix Committed
|
High
|
Matt Riedemann | ||
Queens |
Fix Committed
|
High
|
Matt Riedemann | ||
Ubuntu Cloud Archive |
Fix Released
|
High
|
Unassigned | ||
Ocata |
Fix Released
|
High
|
Unassigned | ||
Pike |
Fix Released
|
High
|
Unassigned | ||
Queens |
Fix Released
|
High
|
Unassigned | ||
Rocky |
Fix Released
|
High
|
Unassigned | ||
nova (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Bionic |
Fix Released
|
High
|
Unassigned | ||
Cosmic |
Fix Released
|
High
|
Unassigned |
Bug Description
It seems like the following commit broke block live migrations of instances with vfat config drive -- https:/
The exception happening is:
May 10 17:01:48 cmp02 nova-compute[4443]: Traceback (most recent call last):
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: listener.cb(fileno)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: result = function(*args, **kwargs)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: return func(*args, **kwargs)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: LOG.error("Live Migration failure: %s", e, instance=instance)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: self.force_
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: six.reraise(
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: bandwidth=
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: destination, params=params, flags=flags)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: result = proxy_call(
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: rv = execute(f, *args, **kwargs)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: six.reraise(c, e, tb)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: rv = meth(*args, **kwargs)
May 10 17:01:48 cmp02 nova-compute[4443]: File "/usr/lib/
May 10 17:01:48 cmp02 nova-compute[4443]: if ret == -1: raise libvirtError ('virDomainMigr
May 10 17:01:48 cmp02 nova-compute[4443]: libvirtError: operation failed: migration of disk vdb failed: Input/output error
May 10 17:01:48 cmp02 nova-compute[4443]: Removing descriptor: 29
(vdb is a vfat config drive)
after turning on the info logging in qemu seen the following in the syslog:
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.204+0000: 10323: info : qemuMonitorSend
May 10 17:01:48 cmp02 libvirtd[10318]: fd=-1
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.204+0000: 10318: info : qemuMonitorIOWr
May 10 17:01:48 cmp02 libvirtd[10318]: len=228 ret=228 errno=0
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.296+0000: 10323: info : qemuMonitorSend
May 10 17:01:48 cmp02 libvirtd[10318]: fd=-1
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.296+0000: 10318: info : qemuMonitorIOWr
May 10 17:01:48 cmp02 libvirtd[10318]: len=228 ret=228 errno=0
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.388+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.388+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.392+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.393+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.395+0000: 10323: error : qemuMigrationDr
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.395+0000: 10323: info : qemuMonitorSend
May 10 17:01:48 cmp02 libvirtd[10318]: fd=-1
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.395+0000: 10318: info : qemuMonitorIOWr
May 10 17:01:48 cmp02 libvirtd[10318]: len=95 ret=95 errno=0
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.398+0000: 10323: info : qemuMonitorSend
May 10 17:01:48 cmp02 libvirtd[10318]: fd=-1
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.398+0000: 10318: info : qemuMonitorIOWr
May 10 17:01:48 cmp02 libvirtd[10318]: len=95 ret=95 errno=0
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.402+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.403+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.414+0000: 10318: info : qemuMonitorJSON
May 10 17:01:48 cmp02 libvirtd[10318]: "wr_operations": 0, "rd_merged": 0, "rd_bytes": 0, "invalid_
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.823+0000: 10323: error : virNetClientPro
May 10 17:01:48 cmp02 libvirtd[10318]: 2018-05-10 17:01:48.845+0000: 10318: error : virNetSocketRea
It seems like qemu's drive mirror expects the full disk size to be present on dest compute, not only what is allocated.
Revert of the patch solved the issue.
description: | updated |
summary: |
- migration of instance with vfat config drive fails + live block migration of instance with vfat config drive fails |
Changed in nova: | |
importance: | Undecided → High |
tags: | added: libvirt live-migration |
Changed in nova: | |
assignee: | Vladyslav Drok (vdrok) → Lee Yarwood (lyarwood) |
Changed in nova (Ubuntu Bionic): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in nova (Ubuntu Cosmic): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in nova (Ubuntu Cosmic): | |
status: | Triaged → Fix Released |
Fix proposed to branch: master /review. openstack. org/567858
Review: https:/