json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Bug #1979649 reported by Felipe Reyes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Glance-Simplestreams-Sync Charm
New
Undecided
Unassigned

Bug Description

This test is unstable and prone to error, it's not clear if this is an actual issue with radosgw, a race condition, or something else.

```
2022-06-23 06:13:14 [INFO] Retrieving product stream information from https://172.16.0.87:443/swift/v1/simplestreams/data//streams/v1/auto.sync.json
2022-06-23 06:14:44 [INFO] ERROR
2022-06-23 06:14:44 [INFO] ======================================================================
2022-06-23 06:14:44 [INFO] ERROR: test_110_local_product_stream (zaza.openstack.charm_tests.glance_simplestreams_sync.tests.GlanceSimpleStreamsSyncTest)
2022-06-23 06:14:44 [INFO] Verify that the local product stream is accessible and has data.
2022-06-23 06:14:44 [INFO] ----------------------------------------------------------------------
2022-06-23 06:14:44 [INFO] Traceback (most recent call last):
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/glance_simplestreams_sync/tests.py", line 125, in test_110_local_product_stream
2022-06-23 06:14:44 [INFO] _check_local_product_streams(expected_images)
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 324, in wrapped_f
2022-06-23 06:14:44 [INFO] return self(f, *args, **kw)
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 404, in __call__
2022-06-23 06:14:44 [INFO] do = self.iter(retry_state=retry_state)
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 349, in iter
2022-06-23 06:14:44 [INFO] return fut.result()
2022-06-23 06:14:44 [INFO] File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
2022-06-23 06:14:44 [INFO] return self.__get_result()
2022-06-23 06:14:44 [INFO] File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
2022-06-23 06:14:44 [INFO] raise self._exception
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 407, in __call__
2022-06-23 06:14:44 [INFO] result = fn(*args, **kwargs)
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/glance_simplestreams_sync/tests.py", line 118, in _check_local_product_streams
2022-06-23 06:14:44 [INFO] product_streams = get_product_streams(url)
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 324, in wrapped_f
2022-06-23 06:14:44 [INFO] return self(f, *args, **kw)
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 404, in __call__
2022-06-23 06:14:44 [INFO] do = self.iter(retry_state=retry_state)
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 360, in iter
2022-06-23 06:14:44 [INFO] raise retry_exc.reraise()
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 193, in reraise
2022-06-23 06:14:44 [INFO] raise self.last_attempt.result()
2022-06-23 06:14:44 [INFO] File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
2022-06-23 06:14:44 [INFO] return self.__get_result()
2022-06-23 06:14:44 [INFO] File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
2022-06-23 06:14:44 [INFO] raise self._exception
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 407, in __call__
2022-06-23 06:14:44 [INFO] result = fn(*args, **kwargs)
2022-06-23 06:14:44 [INFO] File "/home/ubuntu/src/review.opendev.org/openstack/charm-glance-simplestreams-sync/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/glance_simplestreams_sync/tests.py", line 46, in get_product_streams
2022-06-23 06:14:44 [INFO] return json.loads(json_data)
2022-06-23 06:14:44 [INFO] File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
2022-06-23 06:14:44 [INFO] return _default_decoder.decode(s)
2022-06-23 06:14:44 [INFO] File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
2022-06-23 06:14:44 [INFO] obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2022-06-23 06:14:44 [INFO] File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
2022-06-23 06:14:44 [INFO] raise JSONDecodeError("Expecting value", s, err.value) from None
2022-06-23 06:14:44 [INFO] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2022-06-23 06:14:44 [INFO] ----------------------------------------------------------------------
```

ci job results: https://openstack-ci-reports.ubuntu.com/artifacts/7ea/847251/1/check/focal-wallaby/7ea9c01/
patch: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/847251

Felipe Reyes (freyes)
tags: added: unstable-test
Revision history for this message
Felipe Reyes (freyes) wrote :
Revision history for this message
Felipe Reyes (freyes) wrote :

looking at the radosgw logs, we can see a 404 error was returned when auto.sync.json was requested, and we see no matches for that file doing a PUT/POST earlier, so this suggests radosgw is not the one at fault, it should glance-simplestreams-sync, but crashdump doesn't capture the log of the sstream-mirror-glance located at /var/snap/simplestreams/common

2022-06-23T00:05:43.736+0000 7f4066f3d700 1 beast: 0x7f4120265730: 127.0.0.1 - anonymous [23/Jun/2022:00:05:43.728 +0000] "GET /swift/v1/simplestreams/data//streams/v1/auto.sync.json HTTP/1.1" 404 9 - "python-requests/2.28.0" - latency=0.008000043s

Revision history for this message
Felipe Reyes (freyes) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.