metrics server failure missed by tests
Bug #1843315 reported by
Tim Van Steenburgh
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Charmed Kubernetes Testing |
Triaged
|
Wishlist
|
Unassigned |
Bug Description
Opened by hyperbolic2346 on 2019-01-23 16:48:13+00:00 at https:/
-------
We had an issue crop up in 1.11 where the metrics server fails to connect. We need to verify this functionality in some way in our testing. In this specific case, kubelet stopped listening on the port metrics server is trying to reach. This was a known thing in 1.12+ and was fixed in those versions, but 1.11 did not have the fix and this behavior was changed in a point release.
To post a comment you must log in.
We do test metrics in some capacity now:
@pytest. mark.asyncio mark.flaky( max_runs= 5, min_passes=1) metrics( model, tools):
@pytest.
async def test_toggle_
"""Turn metrics on/off via the 'enable-metrics' config on kubernetes-master,
and check that workload status returns to 'active', and that the metrics-server
svc is started and stopped appropriately.
"""
async def check_svc(app, enabled): with_timeout(
verify_ ready,
(unit, "svc", ["metrics-server"], "-n kube-system"),
timeout_ msg="Unable to find metrics-server svc before timeout", with_timeout(
verify_ deleted,
(unit, "svc", "metrics-server", "-n kube-system"),
timeout_ msg="metrics- server svc still exists after timeout",
unit = app.units[0]
if enabled:
await retry_async_
)
else:
await retry_async_
)
app = model.applicati ons["kubernetes -master" ]
k8s_version_str = app.data[ "workload- version" ] minor_version = tuple(int(i) for i in k8s_version_ str.split( ".")[:2] )
click. echo("skipping, k8s version v" + k8s_version_str)
k8s_
if k8s_minor_version < (1, 16):
return
config = await app.get_config() "enable- metrics" ]["value" ]
old_value = config[
new_value = not old_value
await set_config_ and_wait(
app, {"enable-metrics": str(new_value)}, tools, timeout_secs=240
)
await check_svc(app, new_value)
await set_config_ and_wait(
app, {"enable-metrics": str(old_value)}, tools, timeout_secs=240
)
await check_svc(app, old_value)
If this is enough we can close this bug