Create a GPU enable Jupyter Notebook with both Isilon and DirectPV volumes
Failed with error:or:
{
"object": {
"kind": "Notebook",
"namespace": "guillermo-aure",
"name": "kubeflow-deployment-validation",
"uid": "e2deca99-a768-4de4-9274-ef35ff1efb44",
"apiVersion": "kubeflow.org/v1beta1",
"resourceVersion": "90644638"
},
"reason": "FailedCreate",
"message": "Reissued from statefulset/kubeflow-deployment-validation: create Pod kubeflow-deployment-validation-0 in StatefulSet kubeflow-deployment-validation failed error: Internal error occurred: failed calling webhook \"namespace.sidecar-injector.istio.io\": failed to call webhook: Post \"https://istiod.k8s.svc:443/inject?timeout=10s\": service \"istiod\" not found"
It appears the MutatingWebhook is pointing to a service in a non-existent namespace. We see it calling "istiod.k8s.svc", which would be the istiod service in the k8s namespace. I think this should be "istiod.
As a test, we manually removed the "istio-injection: enabled" label on the namespace being tested, and with that removed Notebooks and pods start correctly. So something is misconfigured with the webhook for istio sidecar injection and that is the cause of problems