Can't use DNS for pods in StatefulSet

Bug #1832608 reported by Kenneth Koski
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Critical
Harry Pidcock

Bug Description

I am trying to contact a pod in a StatefulSet created by Juju. According to the docs, `pod-name-0.service-name` should work, but fails for me. I found this bug, which indicates that `spec.serviceName` in the StatefulSet should match `metadata.name` in the Service:

https://github.com/kubernetes/kubernetes/issues/45779

Juju sets up the StatefulSet with a blank `spec.serviceName`. I would test setting that field to match the Service's `metadata.name`, but Kubernetes doesn't allow modifying that field on an existing StatefulSet, and I'm not sure how to tell Juju to set it up like that initially.

Tags: k8s
Revision history for this message
Chris Gregan (cgregan) wrote :

Bumped to field-critical as is blocks HA deployment of OSM which is required for a very important RFQ.

Changed in juju:
assignee: nobody → Ian Booth (wallyworld)
status: New → Triaged
importance: Undecided → Critical
Ian Booth (wallyworld)
Changed in juju:
milestone: none → 2.6.4
assignee: Ian Booth (wallyworld) → Harry Pidcock (hpidcock)
Harry Pidcock (hpidcock)
Changed in juju:
status: Triaged → In Progress
Revision history for this message
Harry Pidcock (hpidcock) wrote :
Revision history for this message
David (davigar15) wrote :
Download full text (4.8 KiB)

Hello, the bug wasn't fixed for me. The service name is still empty. I add the steps to reproduce it:

Fresh machine, Ubuntu Bionic

STEPS:
```
sudo snap install microk8s --classic
sudo snap install juju --classic --channel 2.6/edge
microk8s.status --wait-ready
microk8s.enable dashboard storage dns
sudo alias microk8s.kubectl kubectl
juju bootstrap microk8s
juju add-model osm
juju create-storage-pool operator-storage kubernetes storage-class=microk8s-hostpath
juju create-storage-pool osm-pv kubernetes storage-class=microk8s-hostpath
juju deploy cs:~charmed-osm/mongodb-k8s --channel edge -n 3

# Wait...
juju status --color

Model Controller Cloud/Region Version SLA Timestamp
osm microk8s-localhost microk8s/localhost 2.6.4 unsupported 17:50:42Z

App Version Status Scale Charm Store Rev OS Address Notes
mongodb-k8s active 3 mongodb-k8s jujucharms 12 kubernetes 10.152.183.40

Unit Workload Agent Address Ports Message
mongodb-k8s/0* active idle 10.1.1.10 27017/TCP configured
mongodb-k8s/1 active idle 10.1.1.11 27017/TCP configured
mongodb-k8s/2 active idle 10.1.1.12 27017/TCP configured

kubectl -n osm get all
^[[A^[[A
NAME READY STATUS RESTARTS AGE
pod/mongodb-k8s-0 1/1 Running 0 7m20s
pod/mongodb-k8s-1 1/1 Running 0 7m20s
pod/mongodb-k8s-2 1/1 Running 0 7m20s
pod/mongodb-k8s-operator-0 1/1 Running 0 7m39s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mongodb-k8s ClusterIP 10.152.183.40 <none> 27017/TCP 7m20s

NAME READY AGE
statefulset.apps/mongodb-k8s 3/3 7m20s
statefulset.apps/mongodb-k8s-operator 1/1 7m39s

kubectl -n osm get statefulsets.apps mongodb-k8s -o yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    juju-app-uuid: 190b64ca
    juju.io/controller: 21bea588-f0a5-49e7-8776-f5da0a62ea3e
    juju.io/model: b13f5d92-72ce-4777-86bd-9b320b30f4cd
  creationTimestamp: "2019-06-13T17:45:36Z"
  generation: 1
  name: mongodb-k8s
  namespace: osm
  resourceVersion: "816"
  selfLink: /apis/apps/v1/namespaces/osm/statefulsets/mongodb-k8s
  uid: 0d01c190-8e03-11e9-a2f8-fa163e81b8c7
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      juju-app: mongodb-k8s
  serviceName: ""
  template:
    metadata:
      annotations:
        apparmor.security.beta.kubernetes.io/pod: runtime/default
        juju.io/controller: 21bea588-f0a5-49e7-8776-f5da0a62ea3e
        juju.io/model: b13f5d92-72ce-4777-86bd-9b320b30f4cd
        seccomp.security.beta.kubernetes.io/pod: docker/default
      creationTimestamp: null
      labels:
        juju-app: mongodb-k8s
    spec:
      containers:
      - command:
        - mongod
        - --replSet
        - rs0
        - --bind_ip
        - 0.0.0.0
        env:
        - name: ALLOW_ANONYMOUS_LOGIN
          value: "yes"
        image: registry.jujucharms....

Read more...

Tim Penhey (thumper)
tags: added: k8s
Revision history for this message
Ian Booth (wallyworld) wrote :

I just tested this as per comment 3 and the service name was correctly set.
The only thing I can think of that may explain why it didn't work for you is that there may have been a window where the operator image in docker hub was not yet published and so the juju bootstrap pulled an older version without the patch.

Can you remove and reinstall the snaps and try again?

Also, the steps include the creation of storage pools which are not needed or used in the deploy.

Changed in juju:
status: In Progress → Incomplete
Revision history for this message
David (davigar15) wrote :

The service name is now correctly added.
...
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      juju-app: mongodb-k8s
  serviceName: mongodb-k8s
...

But I am not able to resolve the pods statefulset.

# nslookup mongodb-k8s-0.mongodb-k8s.osm.svc.cluster.local
Server: 10.152.183.10
Address: 10.152.183.10#53

** server can't find mongodb-k8s-0.mongodb-k8s.osm.svc.cluster.local: NXDOMAIN

nslookup mongodb-k8s-0.mongodb-k8s
Server: 10.152.183.10
Address: 10.152.183.10#53

** server can't find mongodb-k8s-0.mongodb-k8s: NXDOMAIN

nslookup mongodb-k8s
Server: 10.152.183.10
Address: 10.152.183.10#53

Name: mongodb-k8s.osm.svc.cluster.local
Address: 10.152.183.173

Revision history for this message
Harry Pidcock (hpidcock) wrote :
Harry Pidcock (hpidcock)
Changed in juju:
status: Incomplete → In Progress
milestone: 2.6.4 → 2.6.5
Harry Pidcock (hpidcock)
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.