live-migration fails because of " Host key verification failed"

It looks like the ssh keys are not set up properly.

The default setup(qemu+ssh) triggers the problem. Finally, live migration successes with qemu+tcp setup in libvirtd configuraiton.

From my perspective, after OpenStack deployment through Autopilot, everything should be all set. Otherwise, there should be some guides telling exact steps needed to perform the operation.

Hi KC, I have hit the same problem, could you provide the detail way to fix it, what do you mean with qemu+tcp setup in libvirtd configuration? thanks.

Below configurations have been used to work around the problem:

 1. Edit /etc/nova/nova.conf on nova nodes which participate in the live migration:

 [libvirt]
 rbd_user =
 rbd_secret_uuid =
 #live_migration_uri = qemu+ssh://%s/system
 live_migration_uri = qemu+tcp://%s/system

 # Disable tunnelled migration so that selective
 # live block migration can be supported.
 live_migration_tunnelled = True

 iscsi_use_multipath = True

 2. Edit /etc/libvirt/libvirtd.conf:
 listen_tcp = 1
 listen_addr = "0.0.0.0"
 auth_tcp = "none"

 3. Restart nova-compute and libvirt:
 # service libvirt-bin restart
 # service nova-compute restart

Meanwhile, such modifications won't work if you are using Ubuntu with kernel 3.x like 3.13.0-100-generic I previously used - it seems there is a kernel bug underground. Latest kernel 4.4.0-45-generic work well with above mentioned configurations.

I'm going to target this at autopilot and the nova-compute charm since this looks to be a config issue instead of a packaging issue.

I have a feeling this is related to bug 1628216 but we'll need to confirm a few details.

For those on the bug impacted, please can you describe the network configuration in your infrastructure; if 'unit-get private-address' does not match the DNS response from nslookup hostname, then we see this type of mismatch and broken live migration due to mismatching SSH keys.

FWIW the solution in #5 might work, but it does disable any level of encryption and security on your compute nodes, allowing anyone with network access to tinkle with the instances running on your hypervisors.

Marking 'Incomplete' for now - if we can confirm the DNS/private-address mismatch problem then we can mark as a dupe of bug 1628216.

James,

I just filed a related bug which narrows down that this is happening if ceph-osd configures ssh key exchanges before nova-compute unit is added to the host.

https://bugs.launchpad.net/charm-nova-compute/+bug/1677707

This is also a Xenial Mitaka cloud, these nodes were added to a juju-deployer environment (Bootstack) manually by adding ubuntu charm to provision host, then ceph-osd add-unit, then nova-compute add-unit.