Mirantis OpenStack

staging for 6.0.1 failed with OSTF error in TestNovaNetwork.test_004_create_servers error

Bug #1455451 reported by Paul Karikh on 2015-05-15

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Mirantis OpenStack	Invalid	High	Timur Nurlygayanov	Mirantis OpenStack 8.0
6.0.x	Won't Fix	High	Timur Nurlygayanov	Mirantis OpenStack 6.0-updates
6.1.x	Invalid	High	Timur Nurlygayanov	Mirantis OpenStack 6.1-updates
7.0.x	Invalid	High	Timur Nurlygayanov	Mirantis OpenStack 7.0
8.0.x	Invalid	High	Timur Nurlygayanov	Mirantis OpenStack 8.0

Bug Description

BVT for CentOS faild here: http://jenkins-product.srt.mirantis.net:8080/view/All/job/6.0.1.test_staging_mirror/143/
The error is 'Flavor for tests was not created. Seems that something is wrong with nova services. Please refer to OpenStack logs for more details.'

2015-05-15 09:07:30,621 - INFO nailgun_client.py:205 -- Run OSTF tests at cluster #1: ['smoke', 'sanity']
2015-05-15 09:07:31,135 - INFO http.py:72 -- self url is http://10.108.5.2:8000
2015-05-15 09:07:31,403 - INFO fuel_web_client.py:570 -- Try to run assert ostf with expected fail name None
2015-05-15 09:07:31,404 - INFO fuel_web_client.py:156 -- Assert OSTF run at cluster #1. Should fail 0 tests named None
2015-05-15 09:07:31,404 - INFO fuel_web_client.py:82 -- Wait OSTF tests at cluster #1 for 1800 seconds
2015-05-15 09:12:48,345 - INFO fuel_web_client.py:184 -- OSTF test statuses are : {u'Create volume and boot instance from it': u'skipped', u'Request snapshot list': u'success', u'Create instance flavor': u'success', u'Request volume list': u'success', u'Create keypair': u'success', u'Request flavor list': u'success', u'Request user list': u'success', u'Launch instance, create snapshot, launch instance from snapshot': u'success', u'Create volume and attach it to instance': u'skipped', u'Request instance list': u'success', u'Create security group': u'success', u'Request stack list': u'success', u'Create user and authenticate with it to Horizon': u'success', u'Request image list': u'success', u'Request active services list': u'success', u'Check network parameters': u'success', u'Launch instance': u'failure', u'Check that required services are running': u'success', u'Check network connectivity from instance via floating IP': u'success', u'Request absolute limits list': u'success'}
2015-05-15 09:12:48,501 - INFO fuel_web_client.py:601 -- Wait for task {u'status': u'running', u'name': u'dump', u'cluster': None, u'result': {}, u'progress': 0, u'message': None, u'id': 10, u'uuid': u'f50c92d7-a35d-40e4-a861-ad8d1db91419'} 300 seconds
2015-05-15 09:14:28,997 - INFO decorators.py:41 -- Saving logs to "/home/jenkins/workspace/6.0.1.staging.centos.bvt_1/logs/fail_error_deploy_neutron_vlan_ha-2015_05_15__09_14_28.tar.gz" file
2015-05-15 09:15:12,780 - INFO decorators.py:125 -- <<<<<****************************************************************************************************>>>>>
2015-05-15 09:15:12,780 - INFO decorators.py:126 -- Failed in method 'deploy_neutron_vlan_ha'. Make snapshot: error_deploy_neutron_vlan_ha
2015-05-15 09:15:12,780 - INFO decorators.py:135 -- You could revert this snapshot using [dos.py revert 6.0.1.staging.CentOS.bvt_1.141.2015-05-15_06-11-45 --snapshot-name error_deploy_neutron_vlan_ha && dos.py resume 6.0.1.staging.CentOS.bvt_1.141.2015-05-15_06-11-45 && virsh net-dumpxml 6.0.1.staging.CentOS.bvt_1.141.2015-05-15_06-11-45_admin | grep -P "(\d+\.){3}" -o | awk '{print "Admin node IP: "$0"2"}']
2015-05-15 09:15:12,780 - INFO decorators.py:140 -- <<<<<****************************************************************************************************>>>>>
FAIL
======================================================================
FAIL: Deploy cluster in HA mode with Neutron VLAN
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/6.0.1.staging.centos.bvt_1/fuelweb_test/helpers/decorators.py", line 60, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/6.0.1.staging.centos.bvt_1/fuelweb_test/tests/test_neutron.py", line 329, in deploy_neutron_vlan_ha
    cluster_id=cluster_id)
  File "/home/jenkins/workspace/6.0.1.staging.centos.bvt_1/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.0.1.staging.centos.bvt_1/fuelweb_test/models/fuel_web_client.py", line 574, in run_ostf
    failed_test_name=failed_test_name)
  File "/home/jenkins/workspace/6.0.1.staging.centos.bvt_1/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.0.1.staging.centos.bvt_1/fuelweb_test/models/fuel_web_client.py", line 197, in assert_ostf_run
    failed_tests_res))
AssertionError: Failed tests, fails: 1 should fail: 0 failed tests name: [{u'Launch instance (failure)': u'Flavor for tests was not created. Seems that something is wrong with nova services. Please refer to OpenStack logs for more details.'}]

See original description

Tags:

Revision history for this message

Paul Karikh (pkarikh) wrote on 2015-05-15:

fail_error_deploy_neutron_vlan_ha-2015_05_15__09_14_28.tar.gz Edit (28.4 MiB, application/x-tar)

Paul Karikh (pkarikh) on 2015-05-15

description:

updated

Revision history for this message

Paul Karikh (pkarikh) wrote on 2015-05-15:

Download full text (4.5 KiB)

Here the Neutron errors from /var/log/docker-logs/ostf.log

2015-05-15 09:08:10.199 21961 DEBUG oslo.db.sqlalchemy.session [req-e63d58a2-60ab-4798-aaeb-4d4d7a3edf95 ] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/session.py:482
    strerror=''.join(err_data).join(out_data))
SSHExecCommandFailed: Command 'ping -q -c1 -w10 10.108.6.130', exit status: 1, Error:
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3015ms
2015-05-15 09:09:24 DEBUG (nmanager) Command 'ping -q -c1 -w10 10.108.6.130', exit status: 1, Error:
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3015ms
2015-05-15 09:10:53 DEBUG (utils) RESP:404 CaseInsensitiveDict({'content-length': '140', 'via': '1.1 node-3.test.domain.local:9696', 'connection': 'close', 'date': 'Fri, 15 May 2015 09:10:53 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-a1b9b616-8087-40a1-a128-f37403cda717'}) {"NeutronError": {"message": "Router [u'7abf2c92-ba97-476d-a1d0-8fcec0872388'] could not be found", "type": "RouterNotFound", "detail": ""}}
2015-05-15 09:10:53 DEBUG (client) Error message: {"NeutronError": {"message": "Router [u'7abf2c92-ba97-476d-a1d0-8fcec0872388'] could not be found", "type": "RouterNotFound", "detail": ""}}
    exception_handler_v20(status_code, des_error_body)
2015-05-15 09:10:53 DEBUG (utils) RESP:404 CaseInsensitiveDict({'content-length': '135', 'via': '1.1 node-3.test.domain.local:9696', 'connection': 'close', 'date': 'Fri, 15 May 2015 09:10:53 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-2a605968-05c9-46a1-920d-3ffe4c0e9742'}) {"NeutronError": {"message": "Subnet 7abf2c92-ba97-476d-a1d0-8fcec0872388 could not be found", "type": "SubnetNotFound", "detail": ""}}
2015-05-15 09:10:53 DEBUG (client) Error message: {"NeutronError": {"message": "Subnet 7abf2c92-ba97-476d-a1d0-8fcec0872388 could not be found", "type": "SubnetNotFound", "detail": ""}}
    exception_handler_v20(status_code, des_error_body)
2015-05-15 09:10:56 DEBUG (utils) RESP:404 CaseInsensitiveDict({'content-length': '137', 'via': '1.1 node-3.test.domain.local:9696', 'connection': 'close', 'date': 'Fri, 15 May 2015 09:10:56 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-6387f76e-a1db-454a-9756-3a4913bf8e0d'}) {"NeutronError": {"message": "Network de13a389-1463-45e6-bc65-143d29ee1d83 could not be found", "type": "NetworkNotFound", "detail": ""}}
2015-05-15 09:10:56 DEBUG (client) Error message: {"NeutronError": {"message": "Network de13a389-1463-45e6-bc65-143d29ee1d83 could not be found", "type": "NetworkNotFound", "detail": ""}}
    exception_handler_v20(status_code, des_error_body)
AssertionError: Flavor for tests was not created. Seems that something is wrong with nova services.
2015-05-15 09:11:03 ERROR (nose_storage_plugin) fuel_health.tests.smoke.test_nova_create_instance_with_connectivity.TestNovaNetwork.test_004_create_servers
AssertionError: Step 2...

Here the Neutron errors from /var/log/docker-logs/ostf.log

And here the error from /Users/paul/Downloads/fuel-snapshot-2015-05-15_10-12-49/10.108.5.2/var/log/docker-logs/remote/nova-api.log
2015-05-15T08:54:06.977236+00:00 debug:  fatal_exception_format_errors  = False
2015-05-15T08:54:07.110387+00:00 debug:  publish_errors                 = False
2015-05-15T09:07:40.548794+00:00 debug:  MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
2015-05-15T09:11:04.040933+00:00 debug:  RESP:404 CaseInsensitiveDict({'date': 'Fri, 15 May 2015 09:11:04 GMT', 'connection': 'close', 'content-type': 'application/json; charset=UTF-8', 'content-length': '146', 'x-openstack-request-id': 'req-c2f5f5c7-47d3-46a0-ab07-20fc01766ccc'}) {"NeutronError": {"message": "Security group c08aa6a5-fddf-4dd5-99fe-3dfd7608a996 does not exist", "type": "SecurityGroupNotFound", "detail": ""}}
2015-05-15T09:11:04.041685+00:00 debug:  Error message: {"NeutronError": {"message": "Security group c08aa6a5-fddf-4dd5-99fe-3dfd7608a996 does not exist", "type": "SecurityGroupNotFound", "detail": ""}}

Revision history for this message

Paul Karikh (pkarikh) wrote on 2015-05-15:

Looks like here https://github.com/stackforge/fuel-ostf/blob/master/fuel_health/nmanager.py#L624 fails because self.find_micro_flavor could be None if cls.compute_client.flavors.list() in OfficialClientTest.find_micro_flavor returns nothing.

So, I think, this error from /10.108.5.2/var/log/docker-logs/remote/node-4.test.domain.local could be relevant.

2015-05-15T09:04:22.978879+00:00 debug: fatal_exception_format_errors = False
2015-05-15T09:04:23.115780+00:00 debug: publish_errors = False
2015-05-15T09:09:05.222657+00:00 debug: Unable to mount image /var/lib/nova/instances/67114f6b-c067-495e-ba83-d8486f462145/disk with error Error mounting /var/lib/nova/instances/67114f6b-c067-495e-ba83-d8486f462145/disk with libguestfs (mount_options: /dev/sda on / (options: ''): mount: you must specify the filesystem type). Cannot resize.
2015-05-15T09:11:30.637322+00:00 debug: Unable to mount image /var/lib/nova/instances/32589638-7a99-408a-8cfb-f5d441143e9b/disk with error Error mounting /var/lib/nova/instances/32589638-7a99-408a-8cfb-f5d441143e9b/disk with libguestfs (mount_options: /dev/sda on / (options: ''): mount: you must specify the filesystem type). Cannot resize.

ruhe (ruhe) on 2015-05-15

Changed in mos:
milestone:	none → 6.0.1
importance:	Undecided → High

Paul Karikh (pkarikh) on 2015-05-15

summary:

- staging job for 6.0.1 failed: Deploy cluster in HA mode with Neutron
- VLAN
+ staging for 6.0.1 failed with OSTF error in
+ TestNovaNetwork.test_004_create_servers error

Paul Karikh (pkarikh) on 2015-05-15

Changed in mos:
assignee:	nobody → MOS Nova (mos-nova)

Revision history for this message

Paul Karikh (pkarikh) wrote on 2015-05-18:

Roman Vyalov suggested to set importrance to Critical because of this bug staging tests are failng and I agree.

Changed in mos:
importance:	High → Critical

Revision history for this message

Timofey Durakov (tdurakov) wrote on 2015-05-18:

There is nothing to do with Nova:
Here is valid schenario code for 6.0: https://github.com/stackforge/fuel-ostf/blob/stable/6.0/fuel_health/nmanager.py#L546
There is attempt to create nano flavor with random id: https://github.com/stackforge/fuel-ostf/blob/stable/6.0/fuel_health/nmanager.py#L275

From nova-api.log:
2015-05-15T09:11:02.572783+00:00 info: 10.108.7.2 "GET /v2/9a32c8806842480196ac94f2d5f70cc8/os-networks HTTP/1.1" status: 200 len: 1491 time: 0.0614278
2015-05-15T09:11:03.690632+00:00 debug: Action: 'create', calling method: <bound method FlavorManageController._create of <nova.api.openstack.compute.contrib.flavormanage.FlavorManageController object at 0x37ff450>>, body: {"flavor": {"vcpus": 1, "disk": 1, "name": "ost1_test-flavor-nano236917154", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "ram": 64, "id": 2200, "swap": 0}}
2015-05-15T09:11:03.702536+00:00 info: HTTP exception thrown: Flavor with ID 2200 already exists.
2015-05-15T09:11:03.703729+00:00 debug: Returning 409 to user: Flavor with ID 2200 already exists.
2015-05-15T09:11:03.706665+00:00 info: 10.108.7.2 "POST /v2/9a32c8806842480196ac94f2d5f70cc8/flavors HTTP/1.1" status: 409 len: 299 time: 0.0226591