Fix Redis cluster tests running in neutron gate

Bug #1645775 reported by Peter Stachowski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
High
Peter Stachowski

Bug Description

With the switch to neutron in the gate, the Redis cluster tests were disabled as they failed fairly frequently with the following error:

    2016-11-24 22:58:13.126973 | 2016-11-24 22:58:13.126 | ERROR: proboscis.case.MethodTest (add_initial_cluster_data)
    2016-11-24 22:58:13.128260 | 2016-11-24 22:58:13.127 | ----------------------------------------------------------------------
    2016-11-24 22:58:13.129735 | 2016-11-24 22:58:13.129 | Traceback (most recent call last):
    2016-11-24 22:58:13.131000 | 2016-11-24 22:58:13.130 | File "/usr/local/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    2016-11-24 22:58:13.132236 | 2016-11-24 22:58:13.131 | compatability.capture_type_error(s_func)
    2016-11-24 22:58:13.133988 | 2016-11-24 22:58:13.133 | File "/usr/local/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    2016-11-24 22:58:13.135165 | 2016-11-24 22:58:13.134 | func()
    2016-11-24 22:58:13.136470 | 2016-11-24 22:58:13.136 | File "/usr/local/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    2016-11-24 22:58:13.137681 | 2016-11-24 22:58:13.137 | func(test_case.state.get_state())
    2016-11-24 22:58:13.139023 | 2016-11-24 22:58:13.138 | File "/opt/stack/new/trove/trove/tests/scenario/groups/cluster_actions_group.py", line 66, in add_initial_cluster_data
    2016-11-24 22:58:13.140312 | 2016-11-24 22:58:13.139 | self.test_runner.run_add_initial_cluster_data()
    2016-11-24 22:58:13.141778 | 2016-11-24 22:58:13.141 | File "/opt/stack/new/trove/trove/tests/scenario/runners/cluster_actions_runners.py", line 180, in run_add_initial_cluster_data
    2016-11-24 22:58:13.143065 | 2016-11-24 22:58:13.142 | self.assert_add_cluster_data(data_type, self.cluster_id)
    2016-11-24 22:58:13.144430 | 2016-11-24 22:58:13.144 | File "/opt/stack/new/trove/trove/tests/scenario/runners/cluster_actions_runners.py", line 187, in assert_add_cluster_data
    2016-11-24 22:58:13.145797 | 2016-11-24 22:58:13.145 | self.test_helper.add_data(data_type, self.extract_ipv4s(cluster.ip)[0])
    2016-11-24 22:58:13.148004 | 2016-11-24 22:58:13.147 | File "/opt/stack/new/trove/trove/tests/scenario/helpers/test_helper.py", line 220, in add_data
    2016-11-24 22:58:13.149252 | 2016-11-24 22:58:13.149 | *args, **kwargs)
    2016-11-24 22:58:13.151050 | 2016-11-24 22:58:13.150 | File "/opt/stack/new/trove/trove/tests/scenario/helpers/test_helper.py", line 265, in _perform_data_action
    2016-11-24 22:58:13.152318 | 2016-11-24 22:58:13.152 | (data_fn_name, self.__class__.__name__, ex))
    2016-11-24 22:58:13.153824 | 2016-11-24 22:58:13.153 | RuntimeError: Error calling add_micro3_data from class RedisHelper - Error calling add_micro3_data from class RedisHelper: CLUSTERDOWN The cluster is down

This will need to be debugged in the gate (probably) and a fix submitted.

Changed in trove:
importance: Undecided → High
milestone: none → next
Revision history for this message
Peter Stachowski (peterstac) wrote :

This may be related to the failure documented in https://bugs.launchpad.net/trove/+bug/1645778

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/409613

Changed in trove:
assignee: nobody → Peter Stachowski (peterstac)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.openstack.org/409613
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=773069cb0b6cd44a02d023ba7a656b383d5e9f5c
Submitter: Jenkins
Branch: master

commit 773069cb0b6cd44a02d023ba7a656b383d5e9f5c
Author: Peter Stachowski <email address hidden>
Date: Thu Oct 13 22:35:23 2016 -0400

    Fix Redis cluster grow scenario tests

    Cluster tests were enabled for Redis as they
    should be working now. The tests were disabled for
    MariaDB (and PXC) until the galera issues are sorted out.

    Support for neutron was added in cluster tests by
    making sure that the proper nic is passed in on
    cluster cre
    ate/grow tests.

    The changes for client non-caching was also implemented.

    The scenario tests were also broken apart into
    sections to facilitate easier manual testing.

    Support for cluster upgrade (currently in review)
    was also added so that when datastore implementations
    are pushed up they can be tested immediately.

    Change-Id: I637cbe30fa42528e8abd2a301baa2ec5048b61fc
    Closes-Bug: 1645775

Changed in trove:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/trove 7.0.0.0b3

This issue was fixed in the openstack/trove 7.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.