openstack_s3 with Swift S3 API does not bootstrap correctly

When bootstrapping on an Openstack network with Swift installed (plus S3 API emulation) and S3_URL set to the Swift service, the bootstrap node does not install correctly. "juju -v status" hangs on "Environment still initializing. Will wait.". The swift container (bucket) appears to be created correctly through the S3 emulation.

SSHing into the bootstrap node, /var/log/juju/machine-agent.log appears to error out with "juju.state.errors.MachineStateNotFound: Machine 0 was not found"

Note that this is limited to "openstack_s3" with Swift plus swift3. "openstack" with Swift works as expected (grabs Swift location from Keystone's object-store), as done "openstack_s3" pointed at a nova-objectstore.

Environment is Swift Folsom, along with the swift3 plugin from https://github.com/fujita/swift3 .

ryan@linda:~$ juju -v bootstrap
2012-10-08 11:44:30,357 DEBUG Initializing juju bootstrap runtime
2012-10-08 11:44:30,366 DEBUG openstack: using auth-mode 'userpass' with https://keystone.example.com:443/v2.0/
2012-10-08 11:44:31,199 DEBUG openstack: authenticated til u'2012-10-09T18:44:32Z'
2012-10-08 11:44:31,199 DEBUG openstack: GET 'https://nova-lcy02.example.com/v2/ACCOUNT/flavors'
2012-10-08 11:44:31,980 DEBUG openstack: 200 '{"flavors": [{"id": "1", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/1", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/1", "rel": "bookmark"}], "name": "m1.tiny"}, {"id": "2", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/2", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/2", "rel": "bookmark"}], "name": "m1.small"}, {"id": "3", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/3", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/3", "rel": "bookmark"}], "name": "m1.medium"}, {"id": "4", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/4", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/4", "rel": "bookmark"}], "name": "m1.large"}, {"id": "5", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/5", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/5", "rel": "bookmark"}], "name": "m1.xlarge"}]}'
2012-10-08 11:44:31,982 INFO Bootstrapping environment 'swifttest1' (origin: distro type: openstack_s3)...
2012-10-08 11:44:32,856 DEBUG Verifying writable storage
2012-10-08 11:44:33,732 DEBUG Launching juju bootstrap instance.
2012-10-08 11:44:33,766 DEBUG access compute @ https://nova-lcy02.example.com/v2/ACCOUNT/os-security-groups
2012-10-08 11:44:33,767 DEBUG openstack: GET 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-groups'
2012-10-08 11:44:34,553 DEBUG openstack: 200 '{"security_groups": [{"rules": [{"from_port": 22, "group": {}, "ip_protocol": "tcp", "to_port": 22, "parent_group_id": 843, "ip_range": {"cidr": "0.0.0.0/0"}, "id": 216}, {"from_port": -1, "group": {}, "ip_protocol": "icmp", "to_port": -1, "parent_group_id": 843, "ip_range": {"cidr": "0.0.0.0/0"}, "id": 217}], "tenant_id": "ACCOUNT", "id": 843, "name": "default", "description": "default"}]}'
2012-10-08 11:44:34,554 DEBUG Creating juju security group juju-swifttest1
2012-10-08 11:44:34,554 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-groups'
2012-10-08 11:44:35,362 DEBUG openstack: 200 '{"security_group": {"rules": [], "tenant_id": "ACCOUNT", "id": 1670, "name": "juju-swifttest1", "description": "juju group for swifttest1"}}'
2012-10-08 11:44:35,363 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-group-rules'
2012-10-08 11:44:36,246 DEBUG openstack: 200 '{"security_group_rule": {"from_port": 22, "group": {}, "ip_protocol": "tcp", "to_port": 22, "parent_group_id": 1670, "ip_range": {"cidr": "0.0.0.0/0"}, "id": 362}}'
2012-10-08 11:44:36,247 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-group-rules'
2012-10-08 11:44:37,163 DEBUG openstack: 200 '{"security_group_rule": {"from_port": 1, "group": {"tenant_id": "ACCOUNT", "name": "juju-swifttest1"}, "ip_protocol": "tcp", "to_port": 65535, "parent_group_id": 1670, "ip_range": {}, "id": 363}}'
2012-10-08 11:44:37,164 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-group-rules'
2012-10-08 11:44:38,053 DEBUG openstack: 200 '{"security_group_rule": {"from_port": 1, "group": {"tenant_id": "ACCOUNT", "name": "juju-swifttest1"}, "ip_protocol": "udp", "to_port": 65535, "parent_group_id": 1670, "ip_range": {}, "id": 364}}'
2012-10-08 11:44:38,054 DEBUG Creating machine security group juju-swifttest1-0
2012-10-08 11:44:38,054 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/os-security-groups'
2012-10-08 11:44:38,947 DEBUG openstack: 200 '{"security_group": {"rules": [], "tenant_id": "ACCOUNT", "id": 1671, "name": "juju-swifttest1-0", "description": "juju group for swifttest1 machine 0"}}'
2012-10-08 11:44:38,947 DEBUG access compute @ https://nova-lcy02.example.com/v2/ACCOUNT/flavors/detail
2012-10-08 11:44:38,947 DEBUG openstack: GET 'https://nova-lcy02.example.com/v2/ACCOUNT/flavors/detail'
2012-10-08 11:44:39,686 DEBUG openstack: 200 '{"flavors": [{"vcpus": 1, "disk": 0, "name": "m1.tiny", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/1", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/1", "rel": "bookmark"}], "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "ram": 512, "id": "1", "swap": ""}, {"vcpus": 1, "disk": 10, "name": "m1.small", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/2", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/2", "rel": "bookmark"}], "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 20, "ram": 2048, "id": "2", "swap": ""}, {"vcpus": 2, "disk": 10, "name": "m1.medium", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/3", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/3", "rel": "bookmark"}], "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 40, "ram": 4096, "id": "3", "swap": ""}, {"vcpus": 4, "disk": 10, "name": "m1.large", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/4", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/4", "rel": "bookmark"}], "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 80, "ram": 8192, "id": "4", "swap": ""}, {"vcpus": 8, "disk": 10, "name": "m1.xlarge", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/flavors/5", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/5", "rel": "bookmark"}], "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 160, "ram": 16384, "id": "5", "swap": ""}]}'
2012-10-08 11:44:39,687 DEBUG access compute @ https://nova-lcy02.example.com/v2/ACCOUNT/servers
2012-10-08 11:44:39,687 DEBUG openstack: POST 'https://nova-lcy02.example.com/v2/ACCOUNT/servers'
2012-10-08 11:44:41,011 DEBUG openstack: 202 '{"server": {"OS-DCF:diskConfig": "MANUAL", "id": "94c92309-7860-44bd-b682-2da23034ffbc", "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc", "rel": "bookmark"}], "adminPass": "REMOVED"}}'
2012-10-08 11:44:42,787 INFO 'bootstrap' command finished successfully

ryan@linda:~$ juju -v status
2012-10-08 11:48:51,501 DEBUG Initializing juju status runtime
2012-10-08 11:48:51,510 DEBUG openstack: using auth-mode 'userpass' with https://keystone.example.com:443/v2.0/
2012-10-08 11:48:51,511 INFO Connecting to environment...
2012-10-08 11:48:53,204 DEBUG openstack: authenticated til u'2012-10-09T18:48:54Z'
2012-10-08 11:48:53,205 DEBUG access compute @ https://nova-lcy02.example.com/v2/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc
2012-10-08 11:48:53,206 DEBUG openstack: GET 'https://nova-lcy02.example.com/v2/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc'
2012-10-08 11:48:54,064 DEBUG openstack: 200 '{"server": {"OS-EXT-STS:task_state": null, "addresses": {"canonistack": [{"version": 4, "addr": "10.55.63.35"}]}, "links": [{"href": "http://nova-lcy02.example.com/v2/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc", "rel": "self"}, {"href": "http://nova-lcy02.example.com/ACCOUNT/servers/94c92309-7860-44bd-b682-2da23034ffbc", "rel": "bookmark"}], "image": {"id": "4326c238-e834-450f-8b5d-29e045d1a4d2", "links": [{"href": "http://nova-lcy02.example.com/ACCOUNT/images/4326c238-e834-450f-8b5d-29e045d1a4d2", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id": "1", "links": [{"href": "http://nova-lcy02.example.com/ACCOUNT/flavors/1", "rel": "bookmark"}]}, "id": "94c92309-7860-44bd-b682-2da23034ffbc", "user_id": "28b21464bfce4233b6f3cdf2d45b30e9", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "status": "ACTIVE", "updated": "2012-10-08T18:46:34Z", "hostId": "4cb114bd10f38d5a3c199af1a504d49d68c0acce6be97c743c1b90c5", "key_name": "", "name": "juju swifttest1 instance 0", "created": "2012-10-08T18:44:41Z", "tenant_id": "ACCOUNT", "metadata": {}}}'
2012-10-08 11:48:54,066 DEBUG Connecting to environment using 10.55.63.35...
2012-10-08 11:48:54,067 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="10.55.63.35" remote_port="2181" local_port="39204".
2012-10-08 11:48:58,583:9323(0x7f16d2f87700):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.6
2012-10-08 11:48:58,583:9323(0x7f16d2f87700):ZOO_INFO@log_env@662: Client environment:host.name=linda
2012-10-08 11:48:58,583:9323(0x7f16d2f87700):ZOO_INFO@log_env@669: Client environment:os.name=Linux
2012-10-08 11:48:58,583:9323(0x7f16d2f87700):ZOO_INFO@log_env@670: Client environment:os.arch=3.5.0-17-generic
2012-10-08 11:48:58,583:9323(0x7f16d2f87700):ZOO_INFO@log_env@671: Client environment:os.version=#27-Ubuntu SMP Fri Oct 5 01:35:26 UTC 2012
2012-10-08 11:48:58,584:9323(0x7f16d2f87700):ZOO_INFO@log_env@679: Client environment:user.name=ryan
2012-10-08 11:48:58,584:9323(0x7f16d2f87700):ZOO_INFO@log_env@687: Client environment:user.home=/home/ryan
2012-10-08 11:48:58,584:9323(0x7f16d2f87700):ZOO_INFO@log_env@699: Client environment:user.dir=/home/ryan
2012-10-08 11:48:58,584:9323(0x7f16d2f87700):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=localhost:39204 sessionTimeout=10000 watcher=0x7f16d2a70700 sessionId=0 sessionPasswd=<null> context=0x35ba180 flags=0
2012-10-08 11:48:58,585:9323(0x7f16ce548700):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:39204]
2012-10-08 11:48:59,889:9323(0x7f16ce548700):ZOO_INFO@check_events@1632: session establishment complete on server [127.0.0.1:39204], sessionId=0x13a41b4f0ca0003, negotiated timeout=10000
2012-10-08 11:49:00,079 DEBUG Environment still initializing. Will wait.

ubuntu@juju-swifttest1-instance-0:~$ cat /var/log/juju/machine-agent.log
2012-10-08 18:48:33,208: twisted@ERROR: Unhandled error in Deferred:
2012-10-08 18:48:33,237: twisted@ERROR: Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 391, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 458, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1095, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/juju/agents/base.py", line 245, in startService
    yield self.start()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/juju/agents/machine.py", line 63, in start
    self.get_machine_id())
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1039, in _inlineCallbacks
    result = g.send(result)
  File "/usr/lib/python2.7/dist-packages/juju/state/machine.py", line 83, in get_machine_state
    raise MachineStateNotFound(machine_id)
juju.state.errors.MachineStateNotFound: Machine 0 was not found