affinity filter not work on master branch

Bug #1770434 reported by Jiang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
High
Unassigned

Bug Description

I deploy openstack with devstack, use master branchs. But the affinity filter not work.

There is only one compute node in my environment. I create a server group with anti-affinity. And then, I create a vm with anti-affinify, the vm create success. But, when I create the second vm, it should be failed, but it success.

I print the spec_obj.instance_group.hosts to log, but I get empty list. So, when I create the second vm, antiaffinity should return False, but return True. This is a problem.

The operations see nova.log.

Revision history for this message
Jiang (jiangpf) wrote :
Revision history for this message
Matt Riedemann (mriedem) wrote :

Tempest tests against the ServerGroup(Anti)AffinityFilters, so I'm not sure what you're seeing here. Can you give a specific set of recreate steps for what you're trying to do?

tags: added: requestspec scheduler
Changed in nova:
status: New → Incomplete
Jiang (jiangpf)
description: updated
Changed in nova:
status: Incomplete → New
Revision history for this message
Matt Riedemann (mriedem) wrote :

@Jiang, I marked this as incomplete because I need more information / answers to my questions in comment 2.

Changed in nova:
status: New → Incomplete
Revision history for this message
Jiang (jiangpf) wrote :
Download full text (12.1 KiB)

[root@greenvm-j14070v2 nova]# openstack image list
+--------------------------------------+--------------------------+--------+
| ID | Name | Status |
+--------------------------------------+--------------------------+--------+
| b11c3607-a353-4d1a-880d-5ee117d19a8a | cirros-0.3.5-x86_64-disk | active |
+--------------------------------------+--------------------------+--------+
[root@greenvm-j14070v2 nova]# openstack flavor list
+----+-----------+-------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------+-------+------+-----------+-------+-----------+
| 1 | m1.tiny | 512 | 1 | 0 | 1 | True |
| 2 | m1.small | 2048 | 20 | 0 | 1 | True |
| 3 | m1.medium | 4096 | 40 | 0 | 2 | True |
| 4 | m1.large | 8192 | 80 | 0 | 4 | True |
| 42 | m1.nano | 64 | 0 | 0 | 1 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | 8 | True |
| 84 | m1.micro | 128 | 0 | 0 | 1 | True |
| c1 | cirros256 | 256 | 0 | 0 | 1 | True |
| d1 | ds512M | 512 | 5 | 0 | 1 | True |
| d2 | ds1G | 1024 | 10 | 0 | 1 | True |
| d3 | ds2G | 2048 | 10 | 0 | 2 | True |
| d4 | ds4G | 4096 | 20 | 0 | 4 | True |
+----+-----------+-------+------+-----------+-------+-----------+
[root@greenvm-j14070v2 nova]# openstack network list
+--------------------------------------+---------+----------------------------------------------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+---------+----------------------------------------------------------------------------+
| dca2e6cc-d3e7-49a5-8a0b-ec17acf9034b | private | 5ccfe577-8128-432c-8980-ca2362ef0ab8, 7dc5ce3e-18fa-417b-8eeb-3e9dc341b481 |
| f6b11b0c-b555-45c2-b0ea-ff5b9d22cbed | public | 398607a4-c906-4807-a667-7c83258aabe0, d101eadc-4092-42f6-8ac9-11d668a0374b |
+--------------------------------------+---------+----------------------------------------------------------------------------+
[root@greenvm-j14070v2 nova]# openstack server group list
+--------------------------------------+---------------+---------------+
| ID | Name | Policies |
+--------------------------------------+---------------+---------------+
| ed739005-fa01-4911-b7be-0f5657a62614 | anti-affinity | anti-affinity |
+--------------------------------------+---------------+---------------+
[root@greenvm-j14070v2 nova]# openstack server create --network dca2e6cc-d3e7-49a5-8a0b-ec17acf9034b --image b11c3607-a353-4d1a-880d-5ee117d19a8a --flavor m1.nano --hint group=ed739005-fa01-4911-b7be-0f5657a62614 cirros-01
+-------------------------------------+-----------------------------------------------------------------+
| Field | Value ...

Changed in nova:
status: Incomplete → New
Revision history for this message
liuzhuangzhuang (liuzhuangzhuang) wrote :

@Jiang, Does the enabled_filters option in the controller node's nova configuration file contains the ServerGroupAntiAffinityFilter?

Revision history for this message
Jiang (jiangpf) wrote :

@liuzhuangzhuang, I deployed openstack with devstack. The enabled_filters is bellow:
enabled_filters = RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

I think I was able to reproduce the problem in a fresh single node devstack [1].

The GroupAntiAffinityFilter printed the following when scheduled the second instance:

Group anti affinity: check if ubuntu not in [] {{(pid=25524) host_passes /opt/stack/nova/nova/scheduler/filters/affinity_filter.py:105}}

This means that the instance_group.hosts field is empty during the second scheduling towards the same host the first instance is already running on [2].

It felt like I'm booting the two instances two quickly and hitting a race. So I waited couple of minutes and tried a 3rd instance. But that also placed to the same (and only) compute host.

[1] http://paste.openstack.org/show/725339/
[2] https://github.com/openstack/nova/blob/fcf406003bcde2b60a515f694f2640e27a9d65fd/nova/scheduler/filters/affinity_filter.py#L101-L105

Changed in nova:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

After some debugging it seems that scheduler_utils.setup_instance_group(context, request_spec) in the conductor [3] returns empty host list because the instance query in [4] returns 0 instances for a non empty uuids filter.

Jul 08 19:43:18 ubuntu nova-conductor[7138]: INFO nova.objects.instance_group [None req-6f7cea32-7573-41ee-8eea-0383c5ade138 admin admin] !!! filters {'deleted': False, 'uuid': [u'bbb51826-a9fa-4b66-b976-e9cf1c75175d', u'e6d4be63-8e4e-4be7-ba41-4b56d234a1b4', u'60eb2895-b098-4d5e-b5b2-d9a0686f39bd', u'4c2abe15-40a1-49f7-ada4-11fa9ad7dfef', u'e869bc16-40f8-4fd1-a738-c3015b999cef', u'eeefa697-95ef-445e-985c-da2ae9e6f9e2', u'65614545-9425-4531-8b17-442c6a18dffd', u'9a2c1e5e-0f84-49de-b42e-7baf7935a139', u'c1e22c58-307a-4a67-8e58-90651d42d8e9', u'6680b18b-6e3b-4934-9679-97a61cee35b2', u'd2ac7593-48e0-4aea-813b-b4133acacc53', u'5085065c-a514-4d2b-8060-9ed12f4cb947', u'45b5c17f-b65f-4e1a-abd7-b5b079a4eb60']}
Jul 08 19:43:18 ubuntu nova-conductor[7138]: INFO nova.objects.instance_group [None req-6f7cea32-7573-41ee-8eea-0383c5ade138 admin admin] !!! instances: 0
Jul 08 19:43:18 ubuntu nova-conductor[7138]: INFO nova.scheduler.utils [None req-6f7cea32-7573-41ee-8eea-0383c5ade138 admin admin] !!! group_hosts set([])
Jul 08 19:43:18 ubuntu nova-conductor[7138]: INFO nova.scheduler.utils [None req-6f7cea32-7573-41ee-8eea-0383c5ade138 admin admin] !!! group_info GroupDetails(hosts=set([]), policies=[u'anti-affinity'], members=[u'bbb51826-a9fa-4b66-b976-e9cf1c75175d', u'e6d4be63-8e4e-4be7-ba41-4b56d234a1b4', u'60eb2895-b098-4d5e-b5b2-d9a0686f39bd', u'4c2abe15-40a1-49f7-ada4-11fa9ad7dfef', u'e869bc16-40f8-4fd1-a738-c3015b999cef', u'eeefa697-95ef-445e-985c-da2ae9e6f9e2', u'65614545-9425-4531-8b17-442c6a18dffd', u'9a2c1e5e-0f84-49de-b42e-7baf7935a139', u'c1e22c58-307a-4a67-8e58-90651d42d8e9', u'6680b18b-6e3b-4934-9679-97a61cee35b2', u'd2ac7593-48e0-4aea-813b-b4133acacc53', u'5085065c-a514-4d2b-8060-9ed12f4cb947', u'45b5c17f-b65f-4e1a-abd7-b5b079a4eb60'])
Jul 08 19:43:18 ubuntu nova-conductor[7138]: INFO nova.conductor.manager [None req-6f7cea32-7573-41ee-8eea-0383c5ade138 admin admin] group members=[u'bbb51826-a9fa-4b66-b976-e9cf1c75175d', u'e6d4be63-8e4e-4be7-ba41-4b56d234a1b4', u'60eb2895-b098-4d5e-b5b2-d9a0686f39bd', u'4c2abe15-40a1-49f7-ada4-11fa9ad7dfef', u'e869bc16-40f8-4fd1-a738-c3015b999cef', u'eeefa697-95ef-445e-985c-da2ae9e6f9e2', u'65614545-9425-4531-8b17-442c6a18dffd', u'9a2c1e5e-0f84-49de-b42e-7baf7935a139', u'c1e22c58-307a-4a67-8e58-90651d42d8e9', u'6680b18b-6e3b-4934-9679-97a61cee35b2', u'd2ac7593-48e0-4aea-813b-b4133acacc53', u'5085065c-a514-4d2b-8060-9ed12f4cb947', u'45b5c17f-b65f-4e1a-abd7-b5b079a4eb60'], group hosts=[]

[3] https://github.com/openstack/nova/blob/c0350da4a1607d7aa113caceaefb5d29303c7eed/nova/conductor/manager.py#L711-L713
[4] https://github.com/openstack/nova/blob/c0350da4a1607d7aa113caceaefb5d29303c7eed/nova/objects/instance_group.py#L422

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

I went back to 18.0.0b1 and still able to reproduce the problem. It feels strange.

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

Regarding the tempest test test_create_server_with_scheduler_hint_group_anti_affinity it does pass but it boots the two instance with a single requests[1]. This means there is a single scheduling attempt not two.

In my devstack the single request case works properly so I conclude that the tempest test does not exercise the same scenario that the reporter and I see failing.

[1] https://github.com/openstack/tempest/blob/bcb6cc6184b6c847f6e72305f52b886835a6ca90/tempest/api/compute/admin/test_servers_on_multinodes.py#L50

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

I pushed the tempest modification that does the two server create in two separate request: https://review.openstack.org/#/c/581349/

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

and the new tempest test reproduces the problem reported in this bug: http://logs.openstack.org/49/581349/1/check/tempest-multinode-full/0ac326a/testr_results.html.gz

Revision history for this message
Matt Riedemann (mriedem) wrote :

This might overlap with bug 1746863.

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

There is a pretty similar bug that is already being worked on https://bugs.launchpad.net/nova/+bug/1746863

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.