resource-get hangs indefinitely on older k8s bundles
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Triaged
|
High
|
Unassigned |
Bug Description
I've not been able to reliably collect any information relating to why this is an issue, however during installing an older version of the kubernetes bundles, resource-get will indefinitely hang halting the deployment until a timeout is reached. In which case the hooks fail, and they start the backoff/retry loop.
To reproduce:
juju add-model $(petname)
juju deploy canonical-
This will deploy an older revision of kubernetes (Which should be in the 1.5.x series). The resources are very different in this release of the charms compared to what's currently in the store for the 1.6.1+ series as we moved to snap packages and redefined the entirety of resources at this time.
This appears to cause a problem with fetching resources on the controller, and can only be worked around by manaully attaching the resource from this revision.
I tested this on JAAS, as well as self hosted controller both with the same end result. The installation process halts during resource-get.
Please advise on if any additional information is required to properly triage and I'll do my best to collect the relevant logs and attach here.
Changed in juju: | |
status: | New → Triaged |
importance: | Undecided → High |
milestone: | none → 2.2.0 |
Changed in juju: | |
milestone: | 2.2.0 → 2.2-rc1 |
Changed in juju: | |
milestone: | 2.2-beta4 → 2.2-rc1 |
The following output is a stackdump issued by sending sigquit (ctrl + \) in a hope it will shed more details on where its failing.
root@ip- 172-31- 1-52:/var/ lib/juju/ agents/ unit-kubernetes -worker- 0/charm# resource-get kubernetes
^\SIGQUIT: quit
PC=0x4792f9 m=0
goroutine 0 [idle]: epollwait( 0x4, 0x7ffd76d16960, 0xffffffff00000080, 0x0, 0xffffffff, 0xc935e0, 0x0, 0x0, 0x0, 0x0, ...)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ runtime/ sys_linux_ amd64.s: 440 +0x19 netpoll( 0x50b5701, 0x0)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ runtime/ netpoll_ epoll.go: 67 +0x94 findrunnable( 0xc82001e000, 0x0)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ runtime/ proc.go: 1955 +0x62c
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ runtime/ proc.go: 2072 +0x24f park_m( 0xc8201dca80)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ runtime/ proc.go: 2137 +0x18b mcall(0x7ffd76d 17070)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ runtime/ asm_amd64. s:233 +0x5b
runtime.
runtime.
runtime.
runtime.schedule()
runtime.
runtime.
goroutine 1 [chan receive]: (*Client) .Call(0xc8202bb 020, 0x334c170, 0xa, 0x2f9cca0, 0xc820480310, 0x21ee300, 0xc8203b0740, 0x0, 0x0)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ net/rpc/ client. go:315 +0xcd 0x7ffd76d17e3d, 0xc, 0xc820345770, 0xc82000a220, 0x2, 0x2, 0x1, 0x0, 0x0)
/tmp/tmp6ruzP3 /juju-core_ 2.1.2/src/ github. com/juju/ juju/cmd/ jujud/main. go:113 +0x36a 0xc82000a220, 0x2, 0x2, 0x0)
/tmp/tmp6ruzP3 /juju-core_ 2.1.2/src/ github. com/juju/ juju/cmd/ jujud/main. go:228 +0x6e5 r(0xc82000a220, 0x2, 0x2)
/tmp/tmp6ruzP3 /juju-core_ 2.1.2/src/ github. com/juju/ juju/cmd/ jujud/main. go:190 +0x35
/tmp/tmp6ruzP3 /juju-core_ 2.1.2/src/ github. com/juju/ juju/cmd/ jujud/main_ nix.go: 22 +0x3b
net/rpc.
main.jujuCMain(
main.Main(
main.MainWrappe
main.main()
goroutine 17 [syscall, locked to thread]:
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ runtime/ asm_amd64. s:1998 +0x1
runtime.goexit()
goroutine 5 [syscall]: signal_ recv(0x0)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ runtime/ sigqueue. go:116 +0x132
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ os/signal/ signal_ unix.go: 22 +0x18
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ os/signal/ signal_ unix.go: 28 +0x37
os/signal.
os/signal.loop()
created by os/signal.init.1
goroutine 25 [IO wait]: pollWait( 0x7f20da56ade0, 0x72, 0xc82048a000)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ runtime/ netpoll. go:160 +0x60 .Wait(0xc820480 300, 0x72, 0x0, 0x0)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ net/fd_ poll_runtime. go:73 +0x3a .WaitRead( 0xc820480300, 0x0, 0x0)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ net/fd_ poll_runtime. go:78 +0x36 .Read(0xc820480 2a0, 0xc82048a000, 0x1000, 0x1000, 0x0, 0x7f20da55a050, 0xc8200121c0)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ net/fd_ unix.go: 250 +0x23a .Read(0xc82002b 108, 0xc82048a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ net/net. go:172 +0xe4 *Reader) .fill(0xc8202ba f60)
/var/lib/ jenkins/ crossbuild/ golang- 1.6/src/ bufio/bufio. go:97 +0x1e9 *Reader) .Read(0xc8202ba f60, 0xc8...
net.runtime_
net.(*pollDesc)
net.(*pollDesc)
net.(*netFD)
net.(*conn)
bufio.(
bufio.(