resource-get hangs indefinitely on older k8s bundles

Bug #1684242 reported by Charles Butler
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
High
Unassigned

Bug Description

I've not been able to reliably collect any information relating to why this is an issue, however during installing an older version of the kubernetes bundles, resource-get will indefinitely hang halting the deployment until a timeout is reached. In which case the hooks fail, and they start the backoff/retry loop.

To reproduce:

juju add-model $(petname)
juju deploy canonical-kubernetes-15

This will deploy an older revision of kubernetes (Which should be in the 1.5.x series). The resources are very different in this release of the charms compared to what's currently in the store for the 1.6.1+ series as we moved to snap packages and redefined the entirety of resources at this time.

This appears to cause a problem with fetching resources on the controller, and can only be worked around by manaully attaching the resource from this revision.

I tested this on JAAS, as well as self hosted controller both with the same end result. The installation process halts during resource-get.

Please advise on if any additional information is required to properly triage and I'll do my best to collect the relevant logs and attach here.

Revision history for this message
Charles Butler (lazypower) wrote :
Download full text (5.1 KiB)

The following output is a stackdump issued by sending sigquit (ctrl + \) in a hope it will shed more details on where its failing.

root@ip-172-31-1-52:/var/lib/juju/agents/unit-kubernetes-worker-0/charm# resource-get kubernetes
^\SIGQUIT: quit
PC=0x4792f9 m=0

goroutine 0 [idle]:
runtime.epollwait(0x4, 0x7ffd76d16960, 0xffffffff00000080, 0x0, 0xffffffff, 0xc935e0, 0x0, 0x0, 0x0, 0x0, ...)
        /var/lib/jenkins/crossbuild/golang-1.6/src/runtime/sys_linux_amd64.s:440 +0x19
runtime.netpoll(0x50b5701, 0x0)
        /var/lib/jenkins/crossbuild/golang-1.6/src/runtime/netpoll_epoll.go:67 +0x94
runtime.findrunnable(0xc82001e000, 0x0)
        /var/lib/jenkins/crossbuild/golang-1.6/src/runtime/proc.go:1955 +0x62c
runtime.schedule()
        /var/lib/jenkins/crossbuild/golang-1.6/src/runtime/proc.go:2072 +0x24f
runtime.park_m(0xc8201dca80)
        /var/lib/jenkins/crossbuild/golang-1.6/src/runtime/proc.go:2137 +0x18b
runtime.mcall(0x7ffd76d17070)
        /var/lib/jenkins/crossbuild/golang-1.6/src/runtime/asm_amd64.s:233 +0x5b

goroutine 1 [chan receive]:
net/rpc.(*Client).Call(0xc8202bb020, 0x334c170, 0xa, 0x2f9cca0, 0xc820480310, 0x21ee300, 0xc8203b0740, 0x0, 0x0)
        /var/lib/jenkins/crossbuild/golang-1.6/src/net/rpc/client.go:315 +0xcd
main.jujuCMain(0x7ffd76d17e3d, 0xc, 0xc820345770, 0xc82000a220, 0x2, 0x2, 0x1, 0x0, 0x0)
        /tmp/tmp6ruzP3/juju-core_2.1.2/src/github.com/juju/juju/cmd/jujud/main.go:113 +0x36a
main.Main(0xc82000a220, 0x2, 0x2, 0x0)
        /tmp/tmp6ruzP3/juju-core_2.1.2/src/github.com/juju/juju/cmd/jujud/main.go:228 +0x6e5
main.MainWrapper(0xc82000a220, 0x2, 0x2)
        /tmp/tmp6ruzP3/juju-core_2.1.2/src/github.com/juju/juju/cmd/jujud/main.go:190 +0x35
main.main()
        /tmp/tmp6ruzP3/juju-core_2.1.2/src/github.com/juju/juju/cmd/jujud/main_nix.go:22 +0x3b

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /var/lib/jenkins/crossbuild/golang-1.6/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
        /var/lib/jenkins/crossbuild/golang-1.6/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
        /var/lib/jenkins/crossbuild/golang-1.6/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /var/lib/jenkins/crossbuild/golang-1.6/src/os/signal/signal_unix.go:28 +0x37

goroutine 25 [IO wait]:
net.runtime_pollWait(0x7f20da56ade0, 0x72, 0xc82048a000)
        /var/lib/jenkins/crossbuild/golang-1.6/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820480300, 0x72, 0x0, 0x0)
        /var/lib/jenkins/crossbuild/golang-1.6/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820480300, 0x0, 0x0)
        /var/lib/jenkins/crossbuild/golang-1.6/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8204802a0, 0xc82048a000, 0x1000, 0x1000, 0x0, 0x7f20da55a050, 0xc8200121c0)
        /var/lib/jenkins/crossbuild/golang-1.6/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82002b108, 0xc82048a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /var/lib/jenkins/crossbuild/golang-1.6/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc8202baf60)
        /var/lib/jenkins/crossbuild/golang-1.6/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc8202baf60, 0xc8...

Read more...

Revision history for this message
Charles Butler (lazypower) wrote :
Download full text (3.6 KiB)

I've also verified its not even making any bins on disk so its hung up in the initial phases of fetching the resource. I don't see any resource paths on disk where I would expect them to be:

root@ip-172-31-1-52:/var/lib/juju/agents/unit-kubernetes-worker-0# tree -L 3
.
├── agent.conf
├── charm
│ ├── actions
│ │ ├── clean-containers
│ │ ├── clean-images
│ │ ├── debug
│ │ ├── microbot
│ │ ├── pause
│ │ └── resume
│ ├── actions.yaml
│ ├── bin
│ │ └── layer_option
│ ├── config.yaml
│ ├── copyright
│ ├── debug-resource-get.txt
│ ├── debug-scripts
│ │ ├── charm-unitdata
│ │ ├── docker
│ │ ├── filesystem
│ │ ├── inotify
│ │ ├── juju-logs
│ │ ├── kubectl
│ │ ├── kubernetes-worker-services
│ │ ├── network
│ │ ├── packages
│ │ ├── sysctl
│ │ └── systemd
│ ├── exec.d
│ │ └── docker-compose
│ ├── HACKING.md
│ ├── hooks
│ │ ├── certificates-relation-broken
│ │ ├── certificates-relation-changed
│ │ ├── certificates-relation-departed
│ │ ├── certificates-relation-joined
│ │ ├── cni-relation-broken
│ │ ├── cni-relation-changed
│ │ ├── cni-relation-departed
│ │ ├── cni-relation-joined
│ │ ├── config-changed
│ │ ├── dockerhost-relation-broken
│ │ ├── dockerhost-relation-changed
│ │ ├── dockerhost-relation-departed
│ │ ├── dockerhost-relation-joined
│ │ ├── hook.template
│ │ ├── install
│ │ ├── kube-api-endpoint-relation-broken
│ │ ├── kube-api-endpoint-relation-changed
│ │ ├── kube-api-endpoint-relation-departed
│ │ ├── kube-api-endpoint-relation-joined
│ │ ├── kube-dns-relation-broken
│ │ ├── kube-dns-relation-changed
│ │ ├── kube-dns-relation-departed
│ │ ├── kube-dns-relation-joined
│ │ ├── leader-elected
│ │ ├── leader-settings-changed
│ │ ├── relations
│ │ ├── sdn-plugin-relation-broken
│ │ ├── sdn-plugin-relation-changed
│ │ ├── sdn-plugin-relation-departed
│ │ ├── sdn-plugin-relation-joined
│ │ ├── start
│ │ ├── stop
│ │ ├── update-status
│ │ └── upgrade-charm
│ ├── icon.svg
│ ├── LAYER_README.md
│ ├── layer.yaml
│ ├── lib
│ │ ├── charms
│ │ └── debug_script.py
│ ├── Makefile
│ ├── metadata.yaml
│ ├── reactive
│ │ ├── docker.py
│ │ ├── __init__.py
│ │ ├── kubernetes_worker.py
│ │ ├── __pycache__
│ │ └── tls_client.py
│ ├── README.md
│ ├── requirements.txt
│ ├── revision
│ ├── scripts
│ │ └── enable_grub_cgroups.sh
│ ├── templates
│ │ ├── default-http-backend.yaml
│ │ ├── docker.defaults
│ │ ├── docker.systemd
│ │ ├── ingress-replication-controller.yaml
│ │ ├── kube-default
│ │ ├── kubelet.defaults
│ │ ├── kubelet.service
│ │ ├── kube-proxy.defaults
│ │ ├── kube-proxy.service
│ │ └── microbot-example.yaml
│ ├── tests
│ │ ├── 10-deploy.py
│ │ └── tests.yaml
│ ├── tox.ini
│ └── wheelhouse
│ ├── charmhelpers-0.10.0.tar.gz
│ ├── charms.docker-0.1.14.tar.gz
│ ├── charms.reactive-0.4.5.tar.gz
│ ├── charms.templating.jinja2-1.0.2.tar.gz
│ ├── Jinja2-2.8.tar.gz
│ ├── MarkupSafe-0.23.tar.g...

Read more...

Changed in juju:
status: New → Triaged
importance: Undecided → High
milestone: none → 2.2.0
Changed in juju:
milestone: 2.2.0 → 2.2-rc1
Changed in juju:
milestone: 2.2-beta4 → 2.2-rc1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.