metrics server failure missed by tests

Bug #1843315 reported by Tim Van Steenburgh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Charmed Kubernetes Testing
Triaged
Wishlist
Unassigned

Bug Description

Opened by hyperbolic2346 on 2019-01-23 16:48:13+00:00 at https://github.com/charmed-kubernetes/jenkins/issues/302

------------------------------------------------------------

We had an issue crop up in 1.11 where the metrics server fails to connect. We need to verify this functionality in some way in our testing. In this specific case, kubelet stopped listening on the port metrics server is trying to reach. This was a known thing in 1.12+ and was fixed in those versions, but 1.11 did not have the fix and this behavior was changed in a point release.

Revision history for this message
Adam Stokes (adam-stokes) wrote :

We do test metrics in some capacity now:

@pytest.mark.asyncio
@pytest.mark.flaky(max_runs=5, min_passes=1)
async def test_toggle_metrics(model, tools):
    """Turn metrics on/off via the 'enable-metrics' config on kubernetes-master,
    and check that workload status returns to 'active', and that the metrics-server
    svc is started and stopped appropriately.

    """

    async def check_svc(app, enabled):
        unit = app.units[0]
        if enabled:
            await retry_async_with_timeout(
                verify_ready,
                (unit, "svc", ["metrics-server"], "-n kube-system"),
                timeout_msg="Unable to find metrics-server svc before timeout",
            )
        else:
            await retry_async_with_timeout(
                verify_deleted,
                (unit, "svc", "metrics-server", "-n kube-system"),
                timeout_msg="metrics-server svc still exists after timeout",
            )

    app = model.applications["kubernetes-master"]

    k8s_version_str = app.data["workload-version"]
    k8s_minor_version = tuple(int(i) for i in k8s_version_str.split(".")[:2])
    if k8s_minor_version < (1, 16):
        click.echo("skipping, k8s version v" + k8s_version_str)
        return

    config = await app.get_config()
    old_value = config["enable-metrics"]["value"]
    new_value = not old_value

    await set_config_and_wait(
        app, {"enable-metrics": str(new_value)}, tools, timeout_secs=240
    )
    await check_svc(app, new_value)

    await set_config_and_wait(
        app, {"enable-metrics": str(old_value)}, tools, timeout_secs=240
    )
    await check_svc(app, old_value)

If this is enough we can close this bug

Revision history for this message
George Kraft (cynerva) wrote :

That test checks to see if the metrics-server Service gets created, but it doesn't really test to make sure metrics are working. It wouldn't cover the case that was originally reported.

Changed in charmed-kubernetes-testing:
importance: Undecided → Wishlist
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.