MOS 5.1 CI RPM tests are failing

Bug #1544514 reported by Alexey Stupnikov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Critical
Alexey Stupnikov

Bug Description

CI team, please investigate the issue with RPM packages testing process in 5.1 CI tests. All tests are failing with the following errors in logs:

2016-02-11 09:37:09,178 - INFO fuel_web_client.py:591 -- Wait for task {u'status': u'running', u'name': u'deploy', u'cluster': 1, u'result': {}, u'progress': 0, u'message': None, u'id': 3, u'uuid': u'13268ef4-7a9c-4384-b2f3-7ebf3aa257bc'} 7800 seconds
2016-02-11 09:49:40,217 - INFO regenerate_repo.py:305 -- Error: Package: openstack-nova-common-2014.1.1-fuel5.1.mira15.git.b7810f1.3cfec2d.noarch (nailgun)
 has unmet dependencies:
2016-02-11 09:49:40,217 - INFO regenerate_repo.py:307 -- Requires: lshell
2016-02-11 09:49:40,264 - INFO fuel_web_client.py:591 -- Wait for task {u'status': u'running', u'name': u'dump', u'cluster': None, u'result': {}, u'progress': 0, u'message': None, u'id': 8, u'uuid': u'21919242-29ab-4d75-a174-c52eda46574e'} 300 seconds
2016-02-11 09:50:05,423 - INFO decorators.py:33 -- Saving logs to "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/logs/fail_error_deploy_simple_flat-2016_02_11__09_50_05.tar.gz" file
2016-02-11 09:51:23,923 - INFO decorators.py:117 -- <<<<<****************************************************************************************************>>>>>
2016-02-11 09:51:23,923 - INFO decorators.py:118 -- Failed in method 'deploy_simple_flat'. Make snapshot: error_deploy_simple_flat
2016-02-11 09:51:23,923 - INFO decorators.py:127 -- You could revert this snapshot using [dos.py revert package_testing-centos-5_1 --snapshot-name error_deploy_simple_flat && dos.py resume package_testing-centos-5_1 && virsh net-dumpxml package_testing-centos-5_1_admin | grep -P "(\d+\.){3}" -o | awk '{print "Admin node IP: "$0"2"}']
2016-02-11 09:51:23,923 - INFO decorators.py:132 -- <<<<<****************************************************************************************************>>>>>
FAIL

======================================================================
FAIL: Deploy cluster in simple mode with flat nova-network
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/helpers/decorators.py", line 52, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/tests/test_simple.py", line 110, in deploy_simple_flat
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/helpers/decorators.py", line 209, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/models/fuel_web_client.py", line 406, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval)
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/models/fuel_web_client.py", line 239, in assert_task_success
    task['status'], 'ready', name=task["name"]
AssertionError: Task 'deploy' has incorrect status. error != ready

----------------------------------------------------------------------
Ran 4 tests in 2180.485s

FAILED (failures=1)
Exception in thread Thread-27 (most likely raised during interpreter shutdown):Exception in thread Thread-175 (most likely raised during interpreter shutdown):Exception in thread Thread-55 (most likely raised during interpreter shutdown):Exception in thread Thread-163 (most likely raised during interpreter shutdown):
 Exception in thread Thread-131 (most likely raised during interpreter shutdown):Exception in thread Thread-253 (most likely raised during interpreter shutdown):Exception in thread Thread-257 (most likely raised during interpreter shutdown):Exception in thread Thread-191 (most likely raised during interpreter shutdown):Exception in thread Thread-39 (most likely raised during interpreter shutdown):Exception in thread Thread-139 (most likely raised during interpreter shutdown):Exception in thread Thread-187 (most likely raised during interpreter shutdown):
 Exception in thread Thread-151 (most likely raised during interpreter shutdown):
 Exception in thread Thread-209 (most likely raised during interpreter shutdown):Exception in thread Thread-51 (most likely raised during interpreter shutdown):Exception in thread Thread-245 (most likely raised during interpreter shutdown):Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_innerException in thread Thread-83 (most likely raised during interpreter shutdown):
 Exception in thread Thread-71 (most likely raised during interpreter shutdown):
 Exception in thread Thread-159 (most likely raised during interpreter shutdown):Exception in thread Thread-127 (most likely raised during interpreter shutdown):
 Exception in thread Thread-277 (most likely raised during interpreter shutdown):

Traceback (most recent call last):Exception in thread Thread-107 (most likely raised during interpreter shutdown):
Traceback (most recent call last):Exception in thread Thread-95 (most likely raised during interpreter shutdown):Exception in thread Thread-235 (most likely raised during interpreter shutdown):Exception in thread Thread-7 (most likely raised during interpreter shutdown):Exception in thread Thread-87 (most likely raised during interpreter shutdown):
Traceback (most recent call last):Traceback (most recent call last):
 Exception in thread Thread-183 (most likely raised during interpreter shutdown):
Traceback (most recent call last):

Traceback (most recent call last):
Exception in thread Thread-241 (most likely raised during interpreter shutdown): File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner

  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/paramiko/transport.py", line 1659, in run File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/paramiko/transport.py", line 1659, in run

<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
 Exception in thread Thread-75 (most likely raised during interpreter shutdown):Exception in thread Thread-261 (most likely raised during interpreter shutdown):Traceback (most recent call last):
Traceback (most recent call last):Exception in thread Thread-143 (most likely raised during interpreter shutdown):
 Exception in thread Thread-135 (most likely raised during interpreter shutdown):Exception in thread Thread-3 (most likely raised during interpreter shutdown):Exception in thread Thread-123 (most likely raised during interpreter shutdown)

Example jobs:
http://osci-jenkins.srt.mirantis.net:8080/job/package_testing_deb_updates/680
http://osci-jenkins.srt.mirantis.net:8080/job/package_testing_deb_updates/681

Reviews to fix:
https://review.fuel-infra.org/#/c/16874/
https://review.fuel-infra.org/#/c/16875
https://review.fuel-infra.org/#/c/16876/

Tags: maintenance
Revision history for this message
Aleksandra Fedorova (bookwar) wrote :

This is a system test failure, I don't see how CI team can help here.

Please investigate with devs/qa

Changed in fuel:
assignee: Fuel CI (fuel-ci) → Alexey Stupnikov (astupnikov)
milestone: none → 5.1.1-updates
Changed in fuel:
status: New → Invalid
Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

I have set status to invalid since this patches were pushed to incorrect branch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.