OpenStack Compute (nova)

Many tests fail when run with random hash seed value

Bug #1350287 reported by Daniel Berrange
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Undecided
Marian Horban

Bug Description

When runing the test suite with a random hash seed, eg

PYTHONHASHSEED=666 ./run_tests.sh

many tests will fail as iterators over hash keys/values change their ordering. For example there are many failures in libvirt like

======================================================================
FAIL: nova.tests.virt.libvirt.test_driver.LibvirtConnTestCase.test_xml_and_uri_rescue_no_ramdisk
----------------------------------------------------------------------
Traceback (most recent call last):
_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
INFO [nova.network.driver] Loading network driver 'nova.network.linux_net'
WARNING [nova.virt.libvirt.firewall] Libvirt module could not be loaded. NWFilterFirewall will not work correctly.
INFO [nova.network.driver] Loading network driver 'nova.network.linux_net'
WARNING [nova.virt.libvirt.driver] Cannot update service status on host: fake-mini,since it is not registered.
}}}

Traceback (most recent call last):
  File "/home/berrange/src/cloud/nova/nova/tests/virt/libvirt/test_driver.py", line 2580, in test_xml_and_uri_rescue_no_ramdisk
    expect_ramdisk=False, rescue=instance_data)
  File "/home/berrange/src/cloud/nova/nova/tests/virt/libvirt/test_driver.py", line 4028, in _check_xml_and_uri
    (check(tree), expected_result, i))
  File "/home/berrange/src/cloud/nova/.venv/lib/python2.7/site-packages/testtools/testcase.py", line 321, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/berrange/src/cloud/nova/.venv/lib/python2.7/site-packages/testtools/testcase.py", line 406, in assertThat
    raise mismatch_error
MismatchError: '8388608' != '2097152': 8388608 != 2097152 failed common check 1

As well as failures across many other tests suites.

This bug is to track resolution of all problems in Nova seen when setting PYTHONHASHSEED to a non-zero value.

Tags: network
Revision history for this message
Daniel Berrange (berrange) wrote :

NB a workaround was applied in this review to force the hash seed to 0 for tox, until the proper fixes are done. https://review.openstack.org/#/c/109700/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/110605

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/110606

Revision history for this message
Daniel Berrange (berrange) wrote :
Download full text (9.4 KiB)

After the two fixes above, I see failures from the following 87 test cases which need resolving

nova.tests.api.ec2.test_cloud.CloudTestCase
    test_describe_instances_with_filters_tags FAIL
nova.tests.api.openstack.compute.contrib.test_availability_zone.AvailabilityZoneApiTest
    test_availability_zone_detail FAIL
nova.tests.api.openstack.compute.contrib.test_flavor_access.FlavorAccessSerializerTest
    test_serializer FAIL
nova.tests.api.openstack.compute.plugins.v3.test_availability_zone.AvailabilityZoneApiTest
    test_availability_zone_detail FAIL
nova.tests.api.openstack.compute.plugins.v3.test_keypairs.KeypairsTest
    test_keypair_create_with_empty_name FAIL
nova.tests.api.openstack.compute.test_versions.VersionsTest
    test_multi_choice_image FAIL
    test_multi_choice_server FAIL
nova.tests.api.openstack.test_xmlutil.SelectorTest
    test_empty_selector FAIL
nova.tests.api.openstack.compute.test_versions.VersionsTest
    test_multi_choice_image_xml FAIL
nova.tests.api.openstack.test_xmlutil.MiscellaneousXMLUtilTests
    test_make_flat_dict FAIL
nova.tests.image.test_glance.TestGlanceSerializer
    test_serialize FAIL
nova.tests.integrated.test_api_samples.AggregatesSampleXmlTest
    test_add_host FAIL
    test_add_metadata FAIL
    test_aggregate_get FAIL
    test_remove_host FAIL
    test_update_aggregate FAIL
nova.tests.integrated.test_api_samples.DiskConfigXmlTest
    test_get_image FAIL
nova.tests.integrated.test_api_samples.DiskConfigJsonTest
    test_get_image FAIL
nova.tests.integrated.test_api_samples.ImageSizeSampleXmlTests
    test_show FAIL
nova.tests.integrated.test_api_samples.ImagesSampleJsonTest
    test_image_get FAIL
nova.tests.integrated.test_api_samples.ImagesSampleXmlTest
    test_image_metadata_post FAIL
nova.tests.integrated.test_api_samples.ImageSizeSampleJsonTests
    test_show FAIL
nova.tests.integrated.test_api_samples.ImagesSampleJsonTest
    test_image_metadata_get FAIL
    test_image_metadata_post FAIL
nova.tests.integrated.test_api_samples.ImagesSampleXmlTest
    test_image_get FAIL
    test_image_meta...

Read more...

Revision history for this message
Clark Boylan (cboylan) wrote :

This bug may be a duplicate of https://bugs.launchpad.net/keystone/+bug/1348818 however tracking nova specifics here is probably easier than on the mega thread bug. So I won't mark this as a duplicate.

Tracy Jones (tjones-i)
tags: added: network
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/112470

Changed in nova:
assignee: nobody → Christopher Yeoh (cyeoh-0)
status: New → In Progress
Revision history for this message
Christopher Yeoh (cyeoh-0) wrote :

https://review.openstack.org/112470 addresses the test_keypair_create_with_empty_name failure

Christopher Yeoh (cyeoh-0)
Changed in nova:
assignee: Christopher Yeoh (cyeoh-0) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/112546

Changed in nova:
assignee: nobody → Christopher Yeoh (cyeoh-0)
Revision history for this message
Christopher Yeoh (cyeoh-0) wrote :

https://review.openstack.org/112546 addresses the test_versions related failures

btw not intending to take the whole bug report, just working through the API related ones as I have the time.

Changed in nova:
assignee: Christopher Yeoh (cyeoh-0) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Daniel Berrange (<email address hidden>) on branch: master
Review: https://review.openstack.org/110605
Reason: No agreement on need of this

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/112546
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8ee037a71fd0e37c94243beeda0273eacc3ba42d
Submitter: Jenkins
Branch: master

commit 8ee037a71fd0e37c94243beeda0273eacc3ba42d
Author: Chris Yeoh <email address hidden>
Date: Thu Aug 7 16:20:03 2014 +0930

    Makes versions API output deterministic

    Makes the order of available versions that the API version
    information is returned to clients deterministic. This is
    necessary to ensure that with PYTHONHASHSEED to fixed to 0 the
    unittests will consistently pass.

    Change-Id: I5b7f6b7f061424adc2263c485562e72c7cc29255
    Partial-Bug: #1350287

OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: nobody → Christopher Yeoh (cyeoh-0)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/112470
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e7840303a683d91b01a8c73328f93419c41d7815
Submitter: Jenkins
Branch: master

commit e7840303a683d91b01a8c73328f93419c41d7815
Author: Chris Yeoh <email address hidden>
Date: Thu Aug 7 14:08:13 2014 +0930

    Make API name validation failure deterministic

    Makes the testing of a valid name in the API deterministic
    with respect to failing when an empty name is supplied
    by the user. Formerly it could fail either of two constraints
    - must be at least one character long
    - does not match the valid regexp which requires one character

    The lack of deterministic behaviour was causing issues with
    some unittests when PYTHONHASHSEED is not set to 0 because
    of the two failure modes. The second constraint for validation
    of names is now modified so it allows empty strings but overall the
    behaviour remains the same because of the first constraint.

    Change-Id: Iea89bb329a8939af5168926717b5c13bdf2a7f62
    Partial-Bug: 1350287

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/110606
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6589a36fbbdd4b0953683d0ba1a34679eb0a540d
Submitter: Jenkins
Branch: master

commit 6589a36fbbdd4b0953683d0ba1a34679eb0a540d
Author: Daniel P. Berrange <email address hidden>
Date: Wed Jul 30 12:02:27 2014 +0100

    tests: fix test_compute to have predictable service list

    The _create_service_entries() method in test_compute.py accepted
    a dict for the list of services. The tests would then pull out
    the first item in the list of keys of the dict. When the python
    hash seed is set to a random value the list of keys has no
    predictable ordering, causing the test suite to break. Changing
    the data structure to use a list of lists instead of a dict
    ensures strict ordering.

    Related-bug: #1350287
    Change-Id: I01a0d4cfb086eb94b49d25b65daf08e55e2087b4

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/144379

Changed in nova:
assignee: Christopher Yeoh (cyeoh-0) → Marian Horban (mhorban)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/144778
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b8885fc6ec3160dc858dce7a21f83ea1d7458118
Submitter: Jenkins
Branch: master

commit b8885fc6ec3160dc858dce7a21f83ea1d7458118
Author: Marian Horban <email address hidden>
Date: Fri Jan 2 10:33:02 2015 -0500

    Expansion of matching XML strings logic

    Extended class XMLMatcher with following features:
    1. Ability to ignore ordering of XML nodes;
    2. Matching version and encoding of XMLs;
    3. Ability to skip empty text nodes(nodes which contain white spaces only)

    Related-Bug: #1350287
    Change-Id: I74b0660d2b09b3c6251ec0eabf37978fa5d03004

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Kevin L. Mitchell (<email address hidden>) on branch: master
Review: https://review.openstack.org/144379
Reason: At my suggestion, XMLMatches was updated to include the desired functionality, but it appears this change didn't get abandoned when the other change went in…

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.