MAAS server gave an unexpected response: 503

Bug #1557241 reported by Nobuto Murata
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
maas-deployer
Confirmed
High
Unassigned

Bug Description

It seems maas-deployer is too early to access API.

2016-03-15 09:52:52,996 INFO Waiting for cloud-init to complete - this usually takes several minutes
2016-03-15 09:58:11,957 INFO done.
2016-03-15 09:58:14,855 ERROR The MAAS server gave an unexpected response: 503

The MAAS server gave an unexpected response: 503

Traceback (most recent call last):
  File "/usr/bin/maas-deployer", line 9, in <module>
    load_entry_point('maas-deployer==0.0.6', 'console_scripts', 'maas-deployer')()
  File "/usr/lib/python2.7/dist-packages/maas_deployer/cli.py", line 88, in main
    engine.deploy(target)
  File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/engine.py", line 71, in deploy
    ssh_user=maas_config['user'])
  File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/maasclient/__init__.py", line 29, in __init__
    self.driver = self._get_driver(api_url, api_key, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/maasclient/__init__.py", line 33, in _get_driver
    return SSHDriver(api_url, api_key, ssh_user=kwargs['ssh_user'])
  File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/maasclient/clidriver.py", line 338, in __init__
    self._login(api_url, api_key)
  File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/maasclient/clidriver.py", line 355, in _login
    execc(cmd, stdin=self.cmd_stdin)
  File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/util.py", line 111, in execc
    output=stderr)
subprocess.CalledProcessError: Command 'ssh -i /home/ubuntu/.ssh/id_maas -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=quiet ubuntu@172.16.X.Y maas
login maas http://172.16.X.Y/MAAS/api/1.0 SECRET' returned non-zero exit status 1

[regiond.log]
2016-03-15 00:58:02 [-] Site starting on 5240
2016-03-15 00:58:02 [-] Factory starting on 5253
2016-03-15 00:58:14 [HTTPChannel,0,127.0.0.1] 127.0.0.1 - - [15/Mar/2016:00:58:14 +0000] "GET
/MAAS/api/1.0/nodegroups/?op=list HTTP/1.1" 503 164 "-" "Python-httplib2/0.8 (gzip)"
2016-03-15 01:02:18 [HTTPChannel,0,127.0.0.1] 127.0.0.1 - - [15/Mar/2016:01:02:18 +0000] "GET
/MAAS/rpc/ HTTP/1.1" 503 164 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-03-15 01:02:18 [maas.websocket.listener] Listening for database notifications.
2016-03-15 01:02:18 [HTTPChannel,0,127.0.0.1] 127.0.0.1 - - [15/Mar/2016:01:02:17 +0000] "GET
/MAAS/rpc/ HTTP/1.1" 503 164 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-03-15 01:02:18 [maas.websocket.listener] Listening for database notifications.
2016-03-15 01:02:18 [maas.websocket.listener] Listening for database notifications.
2016-03-15 01:02:18 [maas.websocket.listener] Listening for database notifications.
2016-03-15 01:02:19 [-] Cluster (master) did not import boot resources; it is not connected to
 the region at this time.
2016-03-15 01:02:20 [-] 127.0.0.1 - - [15/Mar/2016:01:02:20 +0000] "GET /MAAS/rpc/ HTTP/1.1" 2
00 472 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"

[clusterd.log]
2016-03-15 00:58:10+0000 [-] Starting protocol <tftp.protocol.TFTP instance at 0x7f458b572ef0>
2016-03-15 00:58:10+0000 [-] TFTP Listener started at 127.0.0.1:69
2016-03-15 01:02:18+0000 [HTTPPageGetter,client] Region is not advertising RPC endpoints.
2016-03-15 01:02:18+0000 [HTTPPageGetter,client] Region is not advertising RPC endpoints.
2016-03-15 01:02:20+0000 [Uninitialized] ClusterClient connection established (HOST:IPv4Address(TCP, '127.0.0.1', 39922) PEER:IPv4Address(TCP, u'127.0.0.1', 5252))
2016-03-15 01:02:20+0000 [Uninitialized] ClusterClient connection established (HOST:IPv4Address(TCP, '192.168.70.23', 36915) PEER:IPv4Address(TCP, u'192.168.70.23', 5250))
2016-03-15 01:02:20+0000 [ClusterClient,client] Event-loop 'maas:pid=30556' authenticated.
2016-03-15 01:02:20+0000 [Uninitialized] ClusterClient connection established (HOST:IPv4Address(TCP, '127.0.0.1', 45240) PEER:IPv4Address(TCP, u'127.0.0.1', 5251))

Nobuto Murata (nobuto)
summary: - MAAS server gave an unexpected response: 504
+ MAAS server gave an unexpected response: 503
tags: added: cpec
Revision history for this message
Nobuto Murata (nobuto) wrote :

Workaround for now at least for me would be adding the following lines to /usr/lib/python2.7/dist-packages/maas_deployer/vmaas/templates/cloud-init.cfg
====
runcmd:
  - sleep 300
====

Revision history for this message
Edward Hope-Morley (hopem) wrote :

I just hit this, lets get it fixed

Changed in maas-deployer:
status: New → Confirmed
importance: Undecided → High
milestone: none → 0.0.8
Revision history for this message
Nobuto Murata (nobuto) wrote :

FWIW, it happened when it didn't have direct connectivity to Internet. It looks like MAAS waits whatever timeout(5min) on initialization when MAAS cannot access to simplestreams index.

Ante Karamatić (ivoks)
tags: removed: cpec
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.