test_update_access_server_address failing in check-tempest-dsvm-postgres-full

Bug #1258006 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
New
Undecided
Unassigned

Bug Description

Didn't see an existing bug for this but needed to recheck against it:

http://logs.openstack.org/03/57003/11/check/check-tempest-dsvm-postgres-full/5a6a05c/console.html

2013-12-05 01:01:31.307 | Traceback (most recent call last):
2013-12-05 01:01:31.307 | File "tempest/api/compute/servers/test_servers.py", line 94, in test_update_access_server_address
2013-12-05 01:01:31.307 | resp, server = self.create_test_server(wait_until='ACTIVE')
2013-12-05 01:01:31.308 | File "tempest/api/compute/base.py", line 117, in create_test_server
2013-12-05 01:01:31.308 | server['id'], kwargs['wait_until'])
2013-12-05 01:01:31.308 | File "tempest/services/compute/json/servers_client.py", line 160, in wait_for_server_status
2013-12-05 01:01:31.308 | extra_timeout=extra_timeout)
2013-12-05 01:01:31.308 | File "tempest/common/waiters.py", line 73, in wait_for_server_status
2013-12-05 01:01:31.309 | raise exceptions.BuildErrorException(server_id=server_id)
2013-12-05 01:01:31.309 | BuildErrorException: Server aea6135c-a998-470e-b2e1-d8794f7805bf failed to build and is in ERROR status

I do see this in the nova scheduler log:

2013-12-05 00:41:45.588 INFO nova.filters [req-29e72655-eb62-44a9-8149-e031de33a85f ServersTestJSON-tempest-2138013281-user ServersTestJSON-tempest-2138013281-tenant] Filter ComputeFilter returned 0 hosts
2013-12-05 00:41:45.588 WARNING nova.scheduler.driver [req-29e72655-eb62-44a9-8149-e031de33a85f ServersTestJSON-tempest-2138013281-user ServersTestJSON-tempest-2138013281-tenant] [instance: aea6135c-a998-470e-b2e1-d8794f7805bf] Setting instance to ERROR state.

Looks like the ComputeFilter couldn't find a host so it bombs out. I'm not sure why the timestamps don't line up with the failure in console.html.

Looks like this has been showing up since at least 11/22:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRkFJTDogdGVtcGVzdC5hcGkuY29tcHV0ZS5zZXJ2ZXJzLnRlc3Rfc2VydmVycy5TZXJ2ZXJzVGVzdEpTT04udGVzdF91cGRhdGVfYWNjZXNzX3NlcnZlcl9hZGRyZXNzW2dhdGVdXCIgQU5EIGZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImFsbCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJmcm9tIjoiMjAxMy0xMS0yMFQwMjoyMDoxMyswMDowMCIsInRvIjoiMjAxMy0xMi0wNVQwMjozNToxMyswMDowMCIsInVzZXJfaW50ZXJ2YWwiOiIwIn0sInN0YW1wIjoxMzg2MjExNjY5NDEyfQ==

The ComputeFilter itself hasn't changed since 8/21, so nothing interesting there.

This nova scheduling fix merged on 11/21 but I'm not sure why/how that would cause this:

https://github.com/openstack/nova/commit/1d769b0c37f24028bceae568ca66bb0426b28b69

Revision history for this message
Matt Riedemann (mriedem) wrote :

Seeing this right before it fails, indicating the host is down:

2013-12-05 00:41:45.588 DEBUG nova.scheduler.filters.compute_filter [req-29e72655-eb62-44a9-8149-e031de33a85f ServersTestJSON-tempest-2138013281-user ServersTestJSON-tempest-2138013281-tenant] (devstack-precise-check-rax-ord-791311.slave.openstack.org, devstack-precise-check-rax-ord-791311.slave.openstack.org) ram:6930 disk:297984 io_ops:1 instances:7 is disabled or has not been heard from in a while host_passes /opt/stack/new/nova/nova/scheduler/filters/compute_filter.py:44

Phil Day has a patch related to that message: https://review.openstack.org/#/c/58118/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.