juju-core

tools migration fails when upgrading 1.20.14 to 1.24.1 on ec2

Bug #1469130 reported by Andrew Wilkins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
Unassigned
juju-core 1.25-alpha1
1.24
Fix Released
High
Andrew Wilkins
juju-core 1.24.3

Bug Description

When upgrading from 1.20.14 to 1.24.1 on AWS, I found that only machine 0 was upgrading and none of the others. After looking through the logs, I found the following:

2015-06-26 06:56:19 ERROR juju.cmd.jujud upgrade.go:359 upgrade from 1.20.14 to 1.24.1.1 for "machine-0" failed (will retry): migrate tools into environment storage: cannot find tools in provider storage: cannot read product data, invalid URL "https://s3-ap-southeast-2.amazonaws.com/cc7deabc68b34d9d8f6af31786115d29/tools/streams/v1/com.ubuntu.juju:released:tools.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJETX5WILMFNUU5BQ%2F20150626%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-Date=20150626T065619Z&X-Amz-Expires=604800&X-Amz-Signature=7e209533f0df04983cedeb2e60476212cb3d75c1d14181845117b971e55811db&X-Amz-SignedHeaders=host" not found

Turning up logging revealed the reason: "The request signature we calculated does not match the signature you provided. Check your key and signing method."

We made some changes to move everything over to AWS' V4 signing method recently, which may be the cause here.

This can be worked around by upgrading twice. The first upgrade will upgrade machine 0; the second upgrade will upload tools into state/gridfs, so the upgrade step won't get in the way.

Revision history for this message
Andrew Wilkins (axwalk) wrote :

See lp:1466969 for some additional context.

Revision history for this message
Andrew Wilkins (axwalk) wrote :
Download full text (4.2 KiB)

It does actually manage to read index.json, it's com.ubuntu.juju:released:tools.json that causes the problem. Guessing, but I'll say it's to do with colons in the name.

--

2015-06-26 11:55:37 DEBUG juju.environs.simplestreams simplestreams.go:429 read metadata index at "https://s3-ap-southeast-2.amazonaws.com/78ac3a374f2f4c498475863a430c9b1f/tools/streams/v1/index.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJETX5WILMFNUU5BQ%2F20150626%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-Date=20150626T195536Z&X-Amz-Expires=604800&X-Amz-Signature=5155b5f06f78363d3c33c87af4332c64419f432208d8a8492a05b1cd59229af8&X-Amz-SignedHeaders=host"

2015-06-26 11:55:37 TRACE juju.environs.simplestreams simplestreams.go:617 candidate matches for products ["com.ubuntu.juju:centos7:amd64" "com.ubuntu.juju:centos7:i386" "com.ubuntu.juju:centos7:armhf" "com.ubuntu.juju:centos7:arm64" "com.ubuntu.juju:centos7:ppc64el" "com.ubuntu.juju:15.10:amd64" "com.ubuntu.juju:15.10:i386" "com.ubuntu.juju:15.10:armhf" "com.ubuntu.juju:15.10:arm64" "com.ubuntu.juju:15.10:ppc64el" "com.ubuntu.juju:12.04:amd64" "com.ubuntu.juju:12.04:i386" "com.ubuntu.juju:12.04:armhf" "com.ubuntu.juju:12.04:arm64" "com.ubuntu.juju:12.04:ppc64el" "com.ubuntu.juju:win2012hvr2:amd64" "com.ubuntu.juju:win2012hvr2:i386" "com.ubuntu.juju:win2012hvr2:armhf" "com.ubuntu.juju:win2012hvr2:arm64" "com.ubuntu.juju:win2012hvr2:ppc64el" "com.ubuntu.juju:win2012hv:amd64" "com.ubuntu.juju:win2012hv:i386" "com.ubuntu.juju:win2012hv:armhf" "com.ubuntu.juju:win2012hv:arm64" "com.ubuntu.juju:win2012hv:ppc64el" "com.ubuntu.juju:win8:amd64" "com.ubuntu.juju:win8:i386" "com.ubuntu.juju:win8:armhf" "com.ubuntu.juju:win8:arm64" "com.ubuntu.juju:win8:ppc64el" "com.ubuntu.juju:win81:amd64" "com.ubuntu.juju:win81:i386" "com.ubuntu.juju:win81:armhf" "com.ubuntu.juju:win81:arm64" "com.ubuntu.juju:win81:ppc64el" "com.ubuntu.juju:13.04:amd64" "com.ubuntu.juju:13.04:i386" "com.ubuntu.juju:13.04:armhf" "com.ubuntu.juju:13.04:arm64" "com.ubuntu.juju:13.04:ppc64el" "com.ubuntu.juju:13.10:amd64" "com.ubuntu.juju:13.10:i386" "com.ubuntu.juju:13.10:armhf" "com.ubuntu.juju:13.10:arm64" "com.ubuntu.juju:13.10:ppc64el" "com.ubuntu.juju:14.04:amd64" "com.ubuntu.juju:14.04:i386" "com.ubuntu.juju:14.04:armhf" "com.ubuntu.juju:14.04:arm64" "com.ubuntu.juju:14.04:ppc64el" "com.ubuntu.juju:14.10:amd64" "com.ubuntu.juju:14.10:i386" "com.ubuntu.juju:14.10:armhf" "com.ubuntu.juju:14.10:arm64" "com.ubuntu.juju:14.10:ppc64el" "com.ubuntu.juju:win2012r2:amd64" "com.ubuntu.juju:win2012r2:i386" "com.ubuntu.juju:win2012r2:armhf" "com.ubuntu.juju:win2012r2:arm64" "com.ubuntu.juju:win2012r2:ppc64el" "com.ubuntu.juju:12.10:amd64" "com.ubuntu.juju:12.10:i386" "com.ubuntu.juju:12.10:armhf" "com.ubuntu.juju:12.10:arm64" "com.ubuntu.juju:12.10:ppc64el" "com.ubuntu.juju:15.04:amd64" "com.ubuntu.juju:15.04:i386" "com.ubuntu.juju:15.04:armhf" "com.ubuntu.juju:15.04:arm64" "com.ubuntu.juju:15.04:ppc64el" "com.ubuntu.juju:win2012:amd64" "com.ubuntu.juju:win2012:i386" "com.ubuntu.juju:win2012:armhf" "com.ubuntu.juju:win2012:arm64" "com.ubuntu.juju:win2012:ppc64el" "com.ubuntu.juju:win7:amd64" "com.ubuntu.juju:win7:i386" "com.ubuntu.juju:win7:armhf" "com.ub...

Read more...

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Yep, that's it. go-amz will need updating. Seems that AWS has different ideas about how the path should be encoded to Go's net/url package.

Andrew Wilkins (axwalk)
Changed in juju-core:
milestone: 1.24.2 → 1.25.0
Andrew Wilkins (axwalk)
Changed in juju-core:
status: Triaged → In Progress
Andrew Wilkins (axwalk)
Changed in juju-core:
status: In Progress → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.