Glance

ocata - webob.exc.HTTPBadRequest: The Store URI was malformed.

Bug #1649300 reported by Corey Bryant
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
In Progress
Critical
Dharini Chandrasekar

Bug Description

We're hitting the following 2 errors in Ocata Ubuntu packages when running unit tests:

glance.tests.unit.v2.test_images_resource.TestImagesController.test_add_location_possible_on_queued
---------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "glance/tests/unit/v2/test_images_resource.py", line 1607, in test_add_location_possible_on_queued
        output = self.controller.update(request, '1', changes)
      File "glance/common/utils.py", line 363, in wrapped
        return func(self, req, *args, **kwargs)
      File "glance/api/v2/images.py", line 148, in update
        change_method(req, image, change)
      File "glance/api/v2/images.py", line 201, in _do_add
        self._do_add_locations(image, path[1], value)
      File "glance/api/v2/images.py", line 314, in _do_add_locations
        raise webob.exc.HTTPBadRequest(explanation=e.msg)
    webob.exc.HTTPBadRequest: The Store URI was malformed.

glance.tests.unit.v2.test_images_resource.TestImagesController.test_replace_location_possible_on_queued
-------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "glance/tests/unit/v2/test_images_resource.py", line 1587, in test_replace_location_possible_on_queued
        output = self.controller.update(request, '1', changes)
      File "glance/common/utils.py", line 363, in wrapped
        return func(self, req, *args, **kwargs)
      File "glance/api/v2/images.py", line 148, in update
        change_method(req, image, change)
      File "glance/api/v2/images.py", line 182, in _do_replace
        self._do_replace_locations(image, value)
      File "glance/api/v2/images.py", line 288, in _do_replace_locations
        raise webob.exc.HTTPBadRequest(explanation=e.msg)
    webob.exc.HTTPBadRequest: The Store URI was malformed.

This started with introduction of the following commit:

commit 4ac8adbccc2b7bbef3d53f9141079c48f9c768f4
Author: Nikhil Komawar <email address hidden>
Date: Wed Sep 7 17:29:41 2016 -0400

   Restrict location updates to active, queued images

Corey Bryant (corey.bryant)
summary: - webob.exc.HTTPBadRequest: The Store URI was malformed.
+ ocata - webob.exc.HTTPBadRequest: The Store URI was malformed.
Revision history for this message
Ian Cordasco (icordasc) wrote :

Calling this a Critical priority because we're seeing failures in Glance randomly in Zuul and this is causing several problems for merging high priority items in a timely fashion.

Changed in glance:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Ian Cordasco (icordasc) wrote :

http://logs.openstack.org/98/358498/1/check/gate-glance-python34-db/f41067e/testr_results.html.gz

For example is a failure from a few minutes ago

Dharini Chandrasekar (dharini-chandrasekar)
Changed in glance:
assignee: nobody → Dharini Chandrasekar (dharini-chandrasekar)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/409880

Changed in glance:
status: Triaged → In Progress
Revision history for this message
Ian Cordasco (icordasc) wrote :

So looking at the failures from the gate, we're getting BadStoreUri exceptions because glance.quota is trying to make a request and that domain doesn't exist.

The URL we're trying to contact is "http://storeurl.com/container" and that's set as the BASE_URL in glance/tests/unit/utils.py which we import an use in these tests.

What's quite frankly bizarre is why this happens only in those two tests when that BASE_URL is used *everywhere* (which would be a non-issue if we're mocking out the request/connection).

Revision history for this message
Ian Cordasco (icordasc) wrote :

And this is still preventing a critical change from merging: http://logs.openstack.org/98/358498/1/check/gate-glance-python27-db-ubuntu-xenial/f60b787/testr_results.html.gz (another set of results from the same review as before)

Revision history for this message
Dharini Chandrasekar (dharini-chandrasekar) wrote :

While we investigate on this, I am inclined towards temporarily removing these 2 tests for now, to prevent the blocking of other jobs.

The failure of these tests are purely related to test cases and not the actual changes [1] and [2] itself, for which these tests were added/edited.

[1] https://review.openstack.org/#/c/324012/
[2] https://review.openstack.org/#/c/366995/

Revision history for this message
Tony Breeds (o-tony) wrote :

See [1] We're clearly running glance_store code not staying in the FakeStoreAPI()

So it looks to me like sometimes the mocking/faking here isn't effective and we call out to the real glance_store during the unit tests

[1] http://logs.openstack.org/98/358498/1/check/gate-glance-python34-db/fb1944b/console.html#_2016-12-12_21_06_30_907102

Revision history for this message
Ian Cordasco (icordasc) wrote :

> So it looks to me like sometimes the mocking/faking here isn't effective and we call out to the real glance_store during the unit tests

Agreed. Dharini and I came to that conclusion yesterday in IRC but I can't figure out how that's happening. The set up fixtures seem completely reliable. The only thing I could guess would be problematic is mox, but that's a different thing and I can't understand why it would fail transitively.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/409880
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=18053e254a3e2583f293a2d6f0448ec58fc6a9c1
Submitter: Jenkins
Branch: master

commit 18053e254a3e2583f293a2d6f0448ec58fc6a9c1
Author: Dharini Chandrasekar <email address hidden>
Date: Mon Dec 12 18:13:56 2016 +0000

    Skipping tests for location 'add', 'replace' on 'queued' images

    Two of test cases added for the changes [1] and [2]
    have been failing, throwing a webob.exc.HTTPBadRequest due to the
    fake store url "http://storeurl.com/container/fake_location_1"
    being used by both the failing test cases.
    The faking in these tests seem to override the usage of
    FakseStoreApi and instead call glance_store.

    Since, the 2 tests that are failing are caused due to issues
    with the faking and not due to the changes [1] and [2] itself,
    for which they were added, this patch proposes to skip the
    2 failing tests until they are resolved inorder to prevent
    other changes from getting blocked.

    [1] https://review.openstack.org/#/c/366995/
    [2] https://review.openstack.org/#/c/324012/

    Refrence Bug: #1649300

    Change-Id: Ib70ce08528ef92d5fb488e4deb5fe82a7f74a634

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to glance (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/414361

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance (master)

Change abandoned by Abhishek Kekane (<email address hidden>) on branch: master
Review: https://review.openstack.org/414361
Reason: Feel free to restore if required

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.