juju bootstrap fails due to wrong Date header (openstack_s3)

Bug #1096029 reported by Thomas Leonard
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
pyjuju
Triaged
Low
Unassigned

Bug Description

After "juju bootstrap", machine 0 fails to start correctly. /var/log/cloud-init-output.log shows:

usage: juju-admin [-h] [--verbose] [--version] [--log-file LOG_FILE] ...

juju cloud orchestration internal tools

positional arguments:

    initialize Initialize Zookeeper hierarchy

optional arguments:
  -h, --help show this help message and exit
  --verbose, -v Enable verbose logging
  --version show program's version number and exit
  --log-file LOG_FILE, -l LOG_FILE
                        Log output to file
juju-admin: error: unrecognized arguments: version="1.0" encoding="UTF-8"?>^M <Error>^M <Code>AccessDenied</Code>

This is because the curl command failed.

The AccessDenied error comes from /usr/share/pyshared/swift3/middleware.py, which does:

  date = email.utils.parsedate(req.headers['Date'])

However, printing out req.headers['Date'] shows that it is an integer (which time.ctime() shows is exactly 10 years in the future).

I don't know what the correct fix is, but I changed it to ignore unparsable dates and that allowed Juju to work.

I have done two OpenStack installs on two different machines, one starting with Essex and upgrading to Folsom and another starting with Folsom, and I hit this issue both times.

Revision history for this message
Kapil Thangavelu (hazmat) wrote :

does it work with native openstack provider ie not openstack_s3

Revision history for this message
Thomas Leonard (tal-it-innovation) wrote :

Hi Kapil,

The "openstack" provider does not appear to work at all, due to the permissions issues you mentioned previously:

https://lists.ubuntu.com/archives/juju/2012-December/002085.html

Revision history for this message
Kapil Thangavelu (hazmat) wrote :

it sounds like a compatibility issue between swift's s3 middleware and the actual s3 behavior, hence a bug in swift's s3 middleware compatibility. we could try to address in txaws s3 implementation as a workaround, but that's not quite right..

looking over the s3 swift middleware history this is the commit that causes the issue

https://github.com/fujita/swift3/commit/c3341b8de8befc7ed6dfca94de70904baa853c61

its basically disallowing a time delta of more 1m from the server current time, which is broken for this case, in addition to parsing it with the wrong format.

Revision history for this message
Kapil Thangavelu (hazmat) wrote :

err.. disallowing a time delta of more 10m

Curtis Hovey (sinzui)
Changed in juju:
importance: Undecided → Low
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.