Several tempest tests failed because of timed out request

Please, attach Neutron and Nova logs - it can not be investigated without it.

snapshot: https://drive.google.com/a/mirantis.com/file/d/0BxPLDs6wcpbDeVJfb1BWMDhtX0k/view?usp=sharing

Thanks for sharing snapshot!

Unfortunately, snapshot does not contain tempest logs, so comparing time when tests were executed and what has happened at that time is hard.

Does all the tests failed with http://paste.openstack.org/show/593517/?

I found several failures on nova side, for example http://paste.openstack.org/show/593621/, so it will be good if Nova team take a look at this, too.

Also, there was error http://paste.openstack.org/show/593622/ - which means that some of the tests was wrong configured or cleanup does not work properly. I can not say which one was that, as it is not clear what tests were executed.

>>> I found several failures on nova side, for example http://paste.openstack.org/show/593621/, so it will be good if Nova team take a look at this, too.

it must be ok: it's a valid error in Nova when we try to create a live snapshot without support from the guest OS (a qemu agent must be running in the guest), we then retry using a simpler approach - https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1898-L1913

Incomplete - Sofia, please provide complete logs of tempest run so Neutron team could investigate and find out cause of the issue.

Console log looks bad: http://paste.openstack.org/show/594426/ - 2 tests failed because of this.

Other tests failed either with
 - "failed to reach ACTIVE status and task state "None" within the required time (300 s). Current status: BUILD. Current task state: networking."

or:
 - "failed to reach ACTIVE status and task state "None" within the required time (300 s). Current status: BUILD. Current task state: scheduling."

An example of nova-compute log when launching server 3f70c24d-c77d-4cd5-b1c8-d9fd581be98f: http://paste.openstack.org/show/594429/ - so it stops for some reason even before calling to neutron. Probably some general problem with the env.

Nova team please take a look.

Fresh reproduction is available on 172.18.162.134. Master node is 10.109.2.2.

Tempest logs are http://cz7776.bud.mirantis.net:8080/jenkins/job/Tempest_9.x_LVM_SSL/256/artifact/log.log http://cz7776.bud.mirantis.net:8080/jenkins/job/Tempest_9.x_LVM_SSL/256/artifact/tempest.log

Looks like an OVS problem (probably caused by some underlying system problem) on compute node 3: http://paste.openstack.org/show/594509/

Leads to servers being unable to spawn: http://paste.openstack.org/show/594508/

This causes tests to fail with "Server 13babaaa-a1a9-4e82-9141-fcf4bd2e1277 failed to build and is in ERROR status" or "Found multiple IPv4 addresses"

Looks similar to bug https://bugs.launchpad.net/mos/+bug/1606546 (see comment #16)

Agreed with Oleg on this: the latest repro seems to be a different problem with OVS.

I took a look at the original diagnostic snapshot and looks like all the failures are consolidated on the node-3. It's not entirely clear to me what went wrong around 00:52:40 on December 28, but looks like after that nova-compute became buggy, e.g. the periodic tasks were not executed anymore. At the same time atop logs look fine, although there are some weird errors in syslog:

http://paste.openstack.org/show/594532/

Waiting for another repro and a live environment.

It reproduced with https://paste.mirantis.net/show/2936/
Feel free and contact me to take an environment with it

Mail thread started on the OVS issue: https://mail.openvswitch.org/pipermail/ovs-discuss/2017-January/043405.html

It's OVS again and it's stuck in D state:

http://paste.openstack.org/show/594712/
http://paste.openstack.org/show/594713/

I wonder if this may be related:

https://www.spinics.net/lists/kvm/msg142390.html

In https://bugs.launchpad.net/mos/+bug/1652934/comments/7 we saw a level 2 guest (as this lab is deployed on VMs) crashing on do_async_page_fault as well

QA folks, can we give it a try with swap disabled on the host or with L1 guests (i.e. deployed openstack compute nodes) with disabled VMX extension?

MOS Linux, what's your take on this?

@rpodolyaka, it will be checked with nested_kvm disabled

In the last run OVS is not the only process that goes into D state (i.e. the problem is not OVS-specific), now it's mcollective:

http://paste.openstack.org/show/594815/
http://paste.openstack.org/show/594816/

This does not confirm that it's https://www.spinics.net/lists/kvm/msg142390.html for sure, but it very much look like so.

Roman, I'm afraid your fears about https://www.spinics.net/lists/kvm/msg142390.html are true.

There are the following ways to solve the issue:
1. pass 'no-kvmapf' kernel option in L1. This option will disable async pf from L2 to be passed to L0, because it disables it on L1. This will lead to some performance degradation though.
2. there is a kernel patch https://www.spinics.net/lists/kvm/msg142390.html which is not merged (and we can't guess when it will be), and we don't maintain kernel anyway.
3. it's possible though to try to build kvm module (e.g. as dkms) but this might bring more issues than it fixes.

Solution #1 looks best, we're waiting for confirmation that it works.

If it does then we should decide should it be some tunable passed to master node from fuel-qa or some logic in fuel-agent (it generates kernel options for slave nodes). This option can't be passed to any node, because on L0 compute it will lead to performance degradation. Controller nodes will not be affected (no matter HW or VM).

So it seems that setting 'no-kvmapf' on L1 works, tests with that option doesn't fail with any of the symptoms observed earlier in this bug. There are failures anyway, however they doesn't seem related. New failures are reproducable (in two runs it was reproduced) and look like:

-----
    Traceback (most recent call last):
      File "tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc
        return func(*args, **kwargs)
      File "tempest/services/object_storage/container_client.py", line 53, in delete_container
        resp, body = self.delete(url)
      File "tempest/lib/common/rest_client.py", line 307, in delete
        return self.request('DELETE', url, extra_headers, headers, body)
      File "tempest/lib/common/rest_client.py", line 664, in request
        self._error_checker(resp, resp_body)
      File "tempest/lib/common/rest_client.py", line 776, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest.lib.exceptions.Conflict: An object with that identifier already exists
    Details: <html><h1>Conflict</h1><p>There was a conflict when trying to complete your request.</p></html>
-----

More log available [1]

[1] http://paste.openstack.org/show/595048/

Passing to fuel-qa as the solution for this bug requires passing additional option to environment before deploying. This option should enable 'no-kvmapf' option in kernel boot parameters on compute node (L1 guest).

@Dima, unfortunately fuel-qa team doesn't responsible for Tempest testing, so issue was move to mos-qa

@aurlapova, for tests running, the Tempest jobs use environments deployed by fuel-qa scripts, not their owns. And solution which @teselkin-d told about, needs updates on fuel-qa's side exactly.

The problem was not seen on runs for 9.2 snapshots #771 and #776 so far.
And the only reasonable way to implement solution with 'no-kvmapf' option seems to be to add reconfiguration actions on the environment right after the deployment. That can be done within CI that runs tempest suites.