periodic master standalone-full-tempest-api fails during tests- neutron_tempest_plugin.api.test_subnets.SubnetServiceTypeTestJSON

posted skiplist at https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/799432

I think this should be already fixed in Neutron by https://review.opendev.org/c/openstack/neutron/+/797595

Since https://review.opendev.org/c/openstack/neutron/+/797595 is merged and already available in tripleo-ci-testing hash.

In order to test it,
1] We are reverting the skiplist review here: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/799577 and running a testproject here: https://review.rdoproject.org/r/c/testproject/+/34399

I will update the bug once I have the result.

EDIT: https://review.opendev.org/c/openstack/neutron/+/797595 got merged on June 28th, 2021 and
Current tripleo-ci-testing hash for neutron
```
[delorean-component-network]
name=delorean-openstack-neutron-bf7a6061a061bfa08eb1ca96c0ac1bc2d5e580b9
baseurl=https://trunk.rdoproject.org/centos8/component/network/bf/7a/bf7a6061a061bfa08eb1ca96c0ac1bc2d5e580b9_534c9b30
enabled=1
gpgcheck=0
priority=1
```
Above hash pointing to https://opendev.org/openstack/neutron/commit/bf7a6061a061bfa08eb1ca96c0ac1bc2d5e580b9 which contains commits from 19hrs before, so above commit is already in tripleo-ci-testing hash.

So above testproject change https://review.rdoproject.org/r/c/testproject/+/34399 test results will confirm it.

based on the testproject https://review.rdoproject.org/r/c/testproject/+/34399 and https://logserver.rdoproject.org/99/34399/1/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/c17638c/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
```
neutron_tempest_plugin.api.test_subnets.SubnetServiceTypeTestJSON.test_allocate_ips_are_from_correct_subnet[id-7e0edb66-1bb2-4473-ab83-d039cddced0d]
----------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/test_subnets.py", line 91, in test_allocate_ips_are_from_correct_subnet
        cidr=str(cidr_1))
      File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/base.py", line 465, in create_subnet
        **kwargs)
      File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 145, in _create
        resp, body = self.post(uri, post_data)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 299, in post
        return self.request('POST', url, extra_headers, headers, body, chunked)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 703, in request
        self._error_checker(resp, resp_body)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 824, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest.lib.exceptions.Conflict: Conflict with state of target resource
    Details: {'type': 'IpAddressGenerationFailureNoMatchingSubnet', 'message': 'No valid service subnet for the given device owner, network 13fb40e3-9cb7-4e41-beb0-b762f3cb0306, service type network:distributed.', 'detail': ''}

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2021-07-06 08:33:27,115 213754 INFO [tempest.lib.common.rest_client] Request (SubnetServiceTypeTestJSON:test_allocate_ips_are_from_correct_subnet): 409 POST http://192.168.24.3:9696/v2.0/subnets 0.873s
    2021-07-06 08:33:27,116 213754 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: {"subnet": {"network_id": "13fb40e3-9cb7-4e41-beb0-b762f3cb0306", "cidr": "192.168.1.0/24", "ip_version": 4, "service_types": ["test:type_1"], "gateway_ip": "192.168.1.1"}}
        Response - Headers: {'content-type': 'application/json', 'content-length': '232', 'x-openstack-request-id': 'req-b4ce6cfe-fda6-47db-bece-b016abd95a61', 'date': 'Tue, 06 Jul 2021 08:33:27 GMT', 'connection': 'close', 'status': '409', 'content-location': 'http://192.168.24.3:9696/v2.0/subnets'}
            Body: b'{"NeutronError": {"type": "IpAddressGenerationFailureNoMatchingSubnet", "message": "No valid service subnet for the given device owner, network 13fb40e3-9cb7-4e41-beb0-b762f3cb0306, service type network:distributed.", "detail": ""}}'

```

Apart from this patch https://review.opendev.org/c/openstack/neutron/+/7975...

Ok, I think I know what is going on there. It's an issue with how OVN backend works in Neutron. I will try to fix either test or neutron code to make it working

As discussed with slawq on IRC
11:00:07 --> | Tengu (~<email address hidden>) has joined #oooq
11:02:11 chkumar|rover | slaweq: hello, please have a look at this bug https://bugs.launchpad.net/tripleo/+bug/1934622 context:
                          | https://review.opendev.org/c/openstack/neutron/+/797595 is already present in the tested hash, not sure
                          | do we need to make some changes somewhere else?
11:02:34 chkumar|rover | slaweq: https://logserver.rdoproject.org/99/34399/1/check/periodic-tripleo-ci-centos-8-standalone-full-te
                          | mpest-api-master/c17638c/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
11:02:43 chkumar|rover | slaweq: thanks :-)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
11:03:28 slaweq | chkumar|rover: looking
11:08:03 slaweq | chkumar|rover: ok, I think I understand now, it's different issue related to how ovn backend works in
                          | Neutron
11:08:25 slaweq | I will deploy dev vm and will try to fix that today or tomorrow
11:17:38

It seems to be a different issue.

Fix proposed https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/799648

I have tested the above patch here: https://review.rdoproject.org/r/c/testproject/+/34399
and run is green: https://logserver.rdoproject.org/99/34399/2/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/3360bed/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

```
{3} neutron_tempest_plugin.api.test_subnets.SubnetServiceTypeTestJSON.test_allocate_ips_are_from_correct_subnet [4.036313s] ... ok
```
Thank you Slawq for fixing it.

This issue was fixed in the openstack/neutron-tempest-plugin train-last release.