tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON teardown bug

hmm, that's weird, I haven't seen anything like that.

We run refstack-client on all stable releases since Train currently and the test is passing fine, you can see the jobs and their logs f.e. in this review - https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/826888

I wonder what could have created the ports on that particular network.

@mkopec, I'm not sure if this helps or not, but I'm deploying a pretty vanilla OpenStack-Kolla, minimal global changes. We are using some older r710's hardware. Is it potentially a timing issue? If we run the job by itself, it does tear down properly. Only when we run the full suite does it fail.

Hi James, does the test fail every time when you run the full suite? Or does it fail from time to time? I checked again the test and it seems like it passes in the refstack jobs as already pointed out by mkopec.

Hi Lukas, Yes, it fails every time. However, since we originally posted this, we have begun to test Yoga and it does NOT fail there.

Below are the versions we are testing on everything including relevant micro-versions. Please note, that the deployment method is exactly the same on all of these with only minor, non-relevant changes.

REFSTACK_TEST_VERSION = {
    "ussuri": "2020.11",
    "victoria": "2021.11",
    "wallaby": "2021.11",
    "xena": "2021.11",
    "yoga": "2021.11",
}
TEMPEST_VERSION = {
    "ussuri": "29.2.0",
    "victoria": "29.2.0",
    "wallaby": "29.1.0",
    "xena": "31.1.0",
    "yoga": "31.1.0",
}
NOVA_MIN_MICROVERSION = {
    "ussuri": "2.1",
    "victoria": "2.1",
    "wallaby": "2.1",
    "xena": "2.1",
    "yoga": "2.1",
}
NOVA_MAX_MICROVERSION = {
    "ussuri": "2.87",
    "victoria": "2.87",
    "wallaby": "2.88",
    "xena": "2.90",
    "yoga": "2.92",
}

Hi James,

I'm sorry for taking me this long to respond. I can confirm that this bug is valid as I was able to reproduce the issue running the tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON.test_multiple_create [1] test. I think that the issue is caused by this line [2] (the servers are deleted using the servers_client.delete_server which does not wait for the deletion of the servers). I will propose a patch to tempest that should fix this and I will probably propose a patch that flags the test in refstack for now. We will see what others think about this during the review.

[1] https://opendev.org/openstack/tempest/src/commit/cd354d8ce3745a96116d534c1428742c605c61e0/tempest/api/compute/servers/test_multiple_create.py#L26
[2] https://opendev.org/openstack/tempest/src/commit/cd354d8ce3745a96116d534c1428742c605c61e0/tempest/api/compute/servers/test_multiple_create.py#L38

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/875751

Reviewed: https://review.opendev.org/c/openstack/tempest/+/875751
Committed: https://opendev.org/openstack/tempest/commit/d78e405364ca3725820fc7bd4b2f2af198ccc624
Submitter: "Zuul (22348)"
Branch: master

commit d78e405364ca3725820fc7bd4b2f2af198ccc624
Author: Lukas Piwowarski <email address hidden>
Date: Tue Feb 28 15:11:43 2023 +0000

    Fix server cleanup in test_multiple_create test

    The above-mentioned test uses ServersClient.delete_server function to
    delete servers. However, this function does not wait for the successful
    deletion of the servers which sometimes leads to an error (see linked
    bug).

    This patch changes the test so that it uses
    BaseV2ComputeTest.delete_server function which waits for the successful
    deletion of the server.

    Closes-Bug: #1954535
    Change-Id: Ic1ed48fda03ce3c13198d2dd299b57d68b06e63d

This issue was fixed in the openstack/tempest 34.1.0 release.

we've just bumped the tempest version refstack-client uses - https://review.opendev.org/c/openinfra/refstack-client/+/877374 - that tempest version contains the needed fix for this issues. Therefore I'm gonna close this as solved.