nova instance remnant left behind after cold migration completes

Assigning to the distro.openstack PL. Need team to determine if this is expected openstack behavior or if this is a bug that requires nova follow-up.

Gating decision TBD based on assessment

StarlingX is now running against upstream OpenStack Nova master, so this issue is possibly a Nova issue.

Wendy, can you please re-test and check to see if migration back to the original host is impacted by this issue? That would help us determine the Importance of this.

I wasn't able to reproduce this on Nova master. I have vague memories of something like this being reported in older versions, but having been fixed since then, though I can't find hard evidence in the shape of bug reports and/or reviews. I'm going to leave this as Incomplete until we get more information, in the shape of:

- what version of Nova can this be reproduced for?
- more detailed reproducer steps - perhaps a specific storage configuration is needed.

This had failed on hardware lab remote storage system ip 20-27 in the following testcases on the following loads.

FAIL test_cold_migrate_vm[remote-0-0-None-2-volume-confirm]
FAIL test_cold_migrate_vm[remote-1-0-None-1-volume-confirm]
FAIL test_cold_migrate_vm[remote-1-512-None-1-image-confirm]
FAIL test_cold_migrate_vm[remote-0-0-None-2-image_with_vol-confirm]

Lab: IP_20_27
Load: 20190410T013000Z
Job: STX_build_master_master
Build Server: starlingx_mirror
Node Config: 2+4+2
Software Version: 19.01

Lab: IP_20_27
Load: 20190427T013000Z
Job: STX_build_master_master
Build Server: starlingx_mirror
Node Config: 2+4+2
Software Version: 19.01

Fails in the check to see that the source host no longer has instance files after the cold migration

Hosts have the following labels assigned ie. includes remote label
 openstack-compute-node=enabled
 openvswitch=enabled
        sriov=enabled
        remote-storage=enabled

Recent load on 2+3 lab also retested. These test continue to fail for the same reason

BUILD_TYPE="Formal"
BUILD_ID="20190613T013000Z"

test_cold_migrate_vm[remote-0-0-None-2-volume-confirm]
test_cold_migrate_vm[remote-1-0-None-1-volume-confirm]
test_cold_migrate_vm[remote-1-512-None-1-image-confirm]
test_cold_migrate_vm[remote-0-0-None-2-image_with_vol-confirm]

For example
# stat /var/lib/nova/instances/4a6f01a6-d9be-4fab-8524-da901343f39f
  File: ‘/var/lib/nova/instances/4a6f01a6-d9be-4fab-8524-da901343f39f’
  Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd00h/64768d Inode: 13762561 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2019-06-17 19:08:52.056799204 +0000
Modify: 2019-06-17 18:25:43.886821025 +0000
Change: 2019-06-17 18:25:43.886821025 +0000
 Birth: -

It's not explicitly specified above, but the fault scenario involves compute nodes using RBD storage as opposed to local storage.

I reproduced it with StarlingX. After the initial cold migration we're left with this on the original compute node:

compute-1:~$ ls -l /var/lib/nova/instances/
total 36
drwxr-xr-x 2 root root 4096 Jun 19 23:15 _base
-rw-r--r-- 1 root root 32 Jun 19 22:51 compute_nodes
drwxr-xr-x 2 root root 4096 Jun 19 23:19 fba5ba3d-c515-4b5f-aecd-8b9137dcddae
drwxr-xr-x 2 root root 4096 Jun 19 23:15 fba5ba3d-c515-4b5f-aecd-8b9137dcddae_resize
drwxr-xr-x 2 root root 4096 Jun 19 23:15 locks
drwx------ 2 root root 16384 Jun 19 15:08 lost+found

compute-1:/var/lib/nova/instances$ ls -l fba5ba3d-c515-4b5f-aecd-8b9137dcddae
total 0
compute-1:/var/lib/nova/instances$ ls -l fba5ba3d-c515-4b5f-aecd-8b9137dcddae_resize
total 96

After confirming the migration, the "_resize" directory was deleted.

Looking at the timestamps, it appears that something is re-creating the instance directory after it's been renamed.

On each compute node, nova.conf has the following set under the libvirt section:

images_rbd_ceph_conf=/etc/ceph/ceph.conf, images_rbd_pool=ephemeral, images_type=rbd

Setting status to "Confirmed" as per notes from Chris Friesen who is able to reproduce the issue in starlingx. I believe the next step is for the nova team to attempt to reproduce with the specific config above.

Assinging to Yong Hu to follow-up since Bruce is out.

@Martin, which upstream LP was this issue duplicated to?
Or what's the analysis to prove this issue is a general Nova issue instead of a STX issue.

Hi Chris

Have you migrate successfully?

I migrate failed, with such error log in libvirtd
2019-07-16 03:07:55.933+0000: 186884: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor
2019-07-16 03:12:15.290+0000: 186902: warning : qemuDomainObjTaint:7640 : Domain id=6 name='instance-00000009' uuid=c1585fd0-22f9-4b91-ba5e-451cd3a5a6ca is tainted: high-privileges
2019-07-16 03:12:15.667+0000: 186904: error : qemuDomainAgentAvailable:9272 : argument unsupported: QEMU guest agent is not configured

controller-1:/var/lib/nova/instances$ ls
_base c1585fd0-22f9-4b91-ba5e-451cd3a5a6ca compute_nodes locks lost+found
controller-1:/var/lib/nova/instances$

controller-0:~$ ls /var/lib/nova/instances/
_base c1585fd0-22f9-4b91-ba5e-451cd3a5a6ca_resize compute_nodes locks lost+found
controller-0:~$

Still failures in regression
Lab: WCP_113_121
Load: 20190713T013000Z
AssertionError: Instance files found on previous host
FAIL 20190713 16:29:06 test_cold_migrate_vm[remote-0-0-None-2-volume-confirm]
FAIL 20190713 16:31:38 test_cold_migrate_vm[remote-1-0-None-1-volume-confirm]
FAIL 20190713 16:34:23 test_cold_migrate_vm[remote-1-512-None-1-image-confirm]

My reproduce step

1, on active controller
$ openstack --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server create --image 9da41b71-075a-4f28-990c-6fcd570747cd --flavor 39e233dd-acd4-4dfa-bb85-e9f520792ba3 --nic net-id=external-net0 vm0

$ openstack --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server stop 7dd1d8ef-a473-42fd-be61-e2f859f0656a

2, from compute-0 copy folder to /var/lib/nova/instance/<instance uuid>/ compute-1 with same location

3, update database
$ kubectl exec -n openstack -it mariadb-server-0 -- grep password /etc/mysql/admin_user.cnf
to get password
$ kubectl exec -n openstack -it mariadb-server-0 bash
in mariadb container
$ mysql -u root -p <last got mariadb password>
    MariaDB [(none)]> use nova;
    MariaDB [nova]> update instances set host='compute-1', node='compute-1' where uuid='7dd1d8ef-a473-42fd-be61-e2f859f0656a
';
    MariaDB [(none)]> quit;

4, launch vm
openstack --os-username 'admin' --os-password 'Local.123' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server start 7dd1d8ef-a473-42fd-be61-e2f859f0656a

After above stop check nova-compute log
$ kubectl -n openstack logs nova-compute-compute-0-75ea0372-c4l2n -c nova-compute

Got such log nova-compute log
2019-07-19 08:02:50.083 2239894 WARNING nova.compute.manager [req-3df620d7-e138-4259-a98f-687e6c71291d - - - - -] While synchronizing instance power states, found 0 instances in the database and 1 instances on the hypervisor.
2019-07-19 08:03:11.962 2239894 WARNING nova.compute.resource_tracker [req-3df620d7-e138-4259-a98f-687e6c71291d - - - - -] Instance 7dd1d8ef-a473-42fd-be61-e2f859f0656a has been moved to another host compute-1(compute-1). There are allocations remaining against the source host that might need to be removed: {u'resources': {u'MEMORY_MB': 512, u'VCPU': 1, u'DISK_GB': 1}}.

Firstly, please check my reproduce step correct or not. If correct, I think left instances remnants is nova expect behavior.

Nova-compute already could detect instance file in /var/lib/nova/<instance UUID>, so it prompt " 1 instances on the hypervisor" and also find no record in database, "0 instances in the database". Study nova/compute/manager.py, in such case, nova-compute just only promote a WARNING log and do nothing, so if think it is expected behavior.

Chen, can you explain why you manually moved the instance rather than using the nova cold migration? This is causing nova to take very different code paths than would normally be taken on a successful migration.

It is *not* correct behaviour for nova to leave behind files on the source node after a successful cold migration.

My working assumption is that it's something down in the guts of the nova code that behaves differently when configured with "images_type=rbd" that results in these remnants being left behind.

change the priority to low.
If upstream fixes this in future, stx will get it as a part of rebase.

I try to reproduced this problem. But there are no files left after cold migration everytime either using horizon or CIL.

Still seeing this issue in cold migration tests with remote storage using master load 20190727T013000Z, and this has been 100% reproducible.

@hutianhao27
Could you please confirm nova.conf in nova-compute containers has images_type=rbd?
Also horizon or openstack CLIs are not correct tools to reproduce this issue, as mentioned in Test Steps, you need to look at /var/lib/nova/instances/ directory after login to the original compute to see the remnants.

I have reproduced this issue, and there are files left after cold migration. I still try to find out why this issue happened.

There are files left after cold migration every time. But the directory left is empty and has no effect on cold migration. So I think maybe we can delete this empty directory and I'd like to know if this is appropriate。

If the directory left is empty and has no effect on cold migration, I'm wondering that can we think maybe this is not a bug?

@yang, could you help check if the directory is also empty?

@tianhao, Can you double confirm the directory is empty without any files inside?
If only having the empty directory, maybe nova code just doesn't delete the /var/lib/nova/instances/, because of either the permission issue or forgetting to delete it after code migration.

Ok, I will confirm the directory again.

Fix proposed to branch: master
Review: https://review.opendev.org/681652

Fix proposed to branch: master
Review: https://review.opendev.org/682523

Change abandoned by hutianhao27 (hu.tianhao@99cloud.net) on branch: master
Review: https://review.opendev.org/681652
Reason: revert previous patch

I have a patch to solve this problem, but it needs someone to review it.(https://review.opendev.org/#/c/682523/)

Reviewed: https://review.opendev.org/682523
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=404932f82130445472837095b3ad9089c75e2660
Submitter: Zuul
Branch: master

commit 404932f82130445472837095b3ad9089c75e2660
Author: hutianhao27 <hu.tianhao@99cloud.net>
Date: Tue Sep 17 01:56:56 2019 +0000

    Revert "nova shared storage: rbd is always shared storage"

    This reverts commit 05b7f63a42e3512da6fe45d2e6639fb47ed8102b.

    The _is_storage_shared_with method is specifically checking if the
    instance directory is shared. It is not checking if the actual instance
    disks are shared and as a result assumptions cannot be made based on the
    value of images_type.

    Closes-Bug: 1824858
    Change-Id: I52293b6ce3e1ce77fa31b382d0067fb3bc68b21f

Fix proposed to branch: stable/train
Review: https://review.opendev.org/726861

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/726862

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/726863

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/726864

Reviewed: https://review.opendev.org/726861
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=890882ebbf74db14a7c1904cca96cd7f5907493b
Submitter: Zuul
Branch: stable/train

commit 890882ebbf74db14a7c1904cca96cd7f5907493b
Author: hutianhao27 <hu.tianhao@99cloud.net>
Date: Tue Sep 17 01:56:56 2019 +0000

    Revert "nova shared storage: rbd is always shared storage"

    This reverts commit 05b7f63a42e3512da6fe45d2e6639fb47ed8102b.

    The _is_storage_shared_with method is specifically checking if the
    instance directory is shared. It is not checking if the actual instance
    disks are shared and as a result assumptions cannot be made based on the
    value of images_type.

    Closes-Bug: 1824858
    Change-Id: I52293b6ce3e1ce77fa31b382d0067fb3bc68b21f
    (cherry picked from commit 404932f82130445472837095b3ad9089c75e2660)

Reviewed: https://review.opendev.org/726862
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a63c97fd2d13de532330523587894163e32a892f
Submitter: Zuul
Branch: stable/stein

commit a63c97fd2d13de532330523587894163e32a892f
Author: hutianhao27 <hu.tianhao@99cloud.net>
Date: Tue Sep 17 01:56:56 2019 +0000

    Revert "nova shared storage: rbd is always shared storage"

    This reverts commit 05b7f63a42e3512da6fe45d2e6639fb47ed8102b.

    The _is_storage_shared_with method is specifically checking if the
    instance directory is shared. It is not checking if the actual instance
    disks are shared and as a result assumptions cannot be made based on the
    value of images_type.

    Closes-Bug: 1824858
    Change-Id: I52293b6ce3e1ce77fa31b382d0067fb3bc68b21f
    (cherry picked from commit 404932f82130445472837095b3ad9089c75e2660)
    (cherry picked from commit 890882ebbf74db14a7c1904cca96cd7f5907493b)

Reviewed: https://review.opendev.org/726863
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=44609c5847dfe591caf3f40497f1abc7c7aa7384
Submitter: Zuul
Branch: stable/rocky

commit 44609c5847dfe591caf3f40497f1abc7c7aa7384
Author: hutianhao27 <hu.tianhao@99cloud.net>
Date: Tue Sep 17 01:56:56 2019 +0000

    Revert "nova shared storage: rbd is always shared storage"

    This reverts commit 05b7f63a42e3512da6fe45d2e6639fb47ed8102b.

    The _is_storage_shared_with method is specifically checking if the
    instance directory is shared. It is not checking if the actual instance
    disks are shared and as a result assumptions cannot be made based on the
    value of images_type.

    Closes-Bug: 1824858
    Change-Id: I52293b6ce3e1ce77fa31b382d0067fb3bc68b21f
    (cherry picked from commit 404932f82130445472837095b3ad9089c75e2660)
    (cherry picked from commit 890882ebbf74db14a7c1904cca96cd7f5907493b)
    (cherry picked from commit a63c97fd2d13de532330523587894163e32a892f)

Reviewed: https://review.opendev.org/726864
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8ba6d8e9c043cc2682d5bb577c8dc3e49614d15f
Submitter: Zuul
Branch: stable/queens

commit 8ba6d8e9c043cc2682d5bb577c8dc3e49614d15f
Author: hutianhao27 <hu.tianhao@99cloud.net>
Date: Tue Sep 17 01:56:56 2019 +0000

    Revert "nova shared storage: rbd is always shared storage"

    This reverts commit 05b7f63a42e3512da6fe45d2e6639fb47ed8102b.

    The _is_storage_shared_with method is specifically checking if the
    instance directory is shared. It is not checking if the actual instance
    disks are shared and as a result assumptions cannot be made based on the
    value of images_type.

    Closes-Bug: 1824858
    Change-Id: I52293b6ce3e1ce77fa31b382d0067fb3bc68b21f
    (cherry picked from commit 404932f82130445472837095b3ad9089c75e2660)
    (cherry picked from commit 890882ebbf74db14a7c1904cca96cd7f5907493b)
    (cherry picked from commit a63c97fd2d13de532330523587894163e32a892f)
    (cherry picked from commit 44609c5847dfe591caf3f40497f1abc7c7aa7384)

This issue was fixed in the openstack/nova queens-eol release.

This issue was fixed in the openstack/nova rocky-eol release.