Resources can't be downloaded

Bug #1628786 reported by Charles Butler
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Critical
Nate Finch

Bug Description

As a thought experiment I bootstrapped a manual controller, enlisted 2 additional machines and attempted to deploy the Canonical Distribution of Kubernetes.

After deploying the charms from the store, it didn't appear that the resources made it to the charms. I attributed this due to beta characteristics of resources. When attempting to manually attach the resources however this did not function as expected.

$ juju attach kubernetes-master kubernetes=kubernetes-master.tar.gz
ERROR failed to upload resource "kubernetes": PUT https://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/applications/kubernetes-master/resources/kubernetes: Put https://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/applications/kubernetes-master/resources/kubernetes: write tcp 10.0.5.36:45190->198.211.98.33:17070: write: connection reset by peer

$ juju attach kubernetes-master kubernetes=kubernetes-master.tar.gz --debug
01:31:55 INFO juju.cmd supercommand.go:63 running juju [2.0-rc1 gc go1.6.2]
01:31:55 DEBUG juju.cmd supercommand.go:64 args: []string{"juju", "attach", "kubernetes-master", "kubernetes=kubernetes-master.tar.gz", "--debug"}
01:31:55 INFO juju.juju api.go:72 connecting to API addresses: [198.211.98.33:17070 [2604:a880:400:d0::a79:e001]:17070 10.10.0.5:17070 10.136.25.134:17070]
01:31:55 INFO juju.api apiclient.go:507 dialing "wss://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/api"
01:31:55 INFO juju.api apiclient.go:507 dialing "wss://[2604:a880:400:d0::a79:e001]:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/api"
01:31:55 INFO juju.api apiclient.go:507 dialing "wss://10.10.0.5:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/api"
01:31:55 INFO juju.api apiclient.go:507 dialing "wss://10.136.25.134:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/api"
01:31:55 INFO juju.api apiclient.go:302 connection established to "wss://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/api"
01:31:55 DEBUG juju.juju api.go:263 API hostnames unchanged - not resolving
01:31:56 DEBUG httpbakery client.go:244 client do PUT https://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/applications/kubernetes-master/resources/kubernetes {
01:31:56 DEBUG httpbakery client.go:246 } -> error <nil>
01:31:56 DEBUG juju.api apiclient.go:558 health ping failed: connection is shut down
01:31:56 ERROR cmd supercommand.go:458 failed to upload resource "kubernetes": PUT https://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/applications/kubernetes-master/resources/kubernetes: resource#kubernetes-master/kubernetes not found
01:31:56 DEBUG cmd supercommand.go:459 (error details: [{github.com/juju/juju/resource/cmd/upload.go:118: failed to upload resource "kubernetes"} {github.com/juju/juju/resource/cmd/upload.go:132: } {github.com/juju/juju/resource/api/client/client.go:99: } {/build/juju-core-uGAy74/juju-core-2.0-rc1/src/github.com/juju/httprequest/client.go:232: PUT https://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/applications/kubernetes-master/resources/kubernetes} {resource#kubernetes-master/kubernetes not found}])

I was able to upgrade the charm using a a version received from the charm store

charm pull cs:~containers/kubernetes-master-2 and push this to the model.

Afterwords I was able to append the resources. It seems like it just completely missed processing the resources initially, which in turn caused problems.

I don't see anything relevant in the controller logs. Advice on how to further diagnose and report helpful findings would be helpful.

Tags: rteam
Revision history for this message
Charles Butler (lazypower) wrote :
Revision history for this message
Charles Butler (lazypower) wrote :
Changed in juju:
status: New → Triaged
importance: Undecided → Critical
milestone: none → 2.0.0
assignee: nobody → Richard Harding (rharding)
tags: added: teamb
tags: added: rteam
removed: teamb
Curtis Hovey (sinzui)
Changed in juju:
milestone: 2.0-rc3 → 2.0.0
Changed in juju:
assignee: Richard Harding (rharding) → Katherine Cox-Buday (cox-katherine-e)
status: Triaged → In Progress
Curtis Hovey (sinzui)
Changed in juju:
milestone: 2.0.0 → 2.0.1
Curtis Hovey (sinzui)
Changed in juju:
milestone: 2.0.1 → none
Revision history for this message
Nate Finch (natefinch) wrote :

This is not just manual, it's presumably due to a change in the charm store, which is no longer sending the content length.

ERROR juju.resource.resourceadapters (attempt 14) retrying resource download from charm store due to error: no content length found in response

summary: - manually bootstrapped provider not receiving resources
+ Resources can't be downloaded
Revision history for this message
Nate Finch (natefinch) wrote :

Seems to be a charmstore problem, doesn't happen for already-zipped resources like
https://api.jujucharms.com/charmstore/v5/~containers/kubernetes-master-6/resource/kubernetes/4

but does happen with plaintext resources like
https://api.jujucharms.com/charmstore/v5/~natefinch/starsay-4/resource/data/0

An easy test:

go get npf.io/gorram
gorram -i {{.ContentLength}} net/http Get <url>

Should print the content length, but for resources that aren't gzipped, it prints -1.

you can also use

gorram -i '{{printf "%#v" .}}' net/http Get <url>

to show the full response data (minus body)

Note that curl seems to do some extra processing to get the correct content-length, even though it's not actually in the response.

Changed in juju:
assignee: Katherine Cox-Buday (cox-katherine-e) → Nate Finch (natefinch)
Revision history for this message
Nate Finch (natefinch) wrote :

in conversation with the charmstore team, it seems like this is an expected response. We don't technically need to check the content-length, since we can check the fingerprint, and that would change if the length changed.

Revision history for this message
Nate Finch (natefinch) wrote :
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
milestone: none → 2.1-beta2
Curtis Hovey (sinzui)
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.