release test 1.23 juju bootstrapping on top of kubernetes fails due to AWS volume timeout

For release testing 1.23, bootstrapping controller on k8s fails:

-----------------
Creating Juju controller "foundations-k8s" on kubernetes_cloud/us-east-1
Bootstrap to Kubernetes cluster identified as ec2/us-east-1
Fetching Juju Dashboard 0.8.1
Creating k8s resources for controller "controller-foundations-k8s"
ERROR failed to bootstrap model: creating controller stack: creating statefulset for controller: timed out waiting for controller pod: pending: -

2021-12-14-11:11:19 root ERROR juju controller is bootstrapped but unreachable.
Traceback (most recent call last):
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/juju_cli.py", line 171, in bootstrap
    local(command, output_mode="live")
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/remotehelpers.py", line 249, in local
    return run_cmd(cmd, target_machine, **kwargs)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/remotehelpers.py", line 182, in run_cmd
    raise subprocess.CalledProcessError(ps.returncode, cmd, output=out, stderr=err)
subprocess.CalledProcessError: Command '['juju', 'bootstrap', '--config', 'bootstrap-timeout=1800', '--model-default', '/home/ubuntu/project/generated/model_defaults.yaml', 'kubernetes_cloud', 'foundations-k8s']' returned non-zero exit status 1.
-----------------

No problem found in the kubernetes layer, kubernetes validation passes.

Crash dumps & artifacts:
https://oil-jenkins.canonical.com/artifacts/610a3f4e-275b-41a5-99cd-d35c60fbacf7/index.html
List of testruns with this problem:
https://solutions.qa.canonical.com/bugs/bugs/bug/1954807

We will do some debugging with sqa this afternoon.