maas-deployer failed ValueError: No JSON object could be decoded

http://bazaar.launchpad.net/~maas-deployers/maas-deployer/next/revision/78

backport in progress...

NOTE: to fix this we are basically skipping the images import check until we have a proper workaround. As a result, if commissioning is started before image are fully imported it might fail. The simple solution to this is to go into the UI and restart the commission (ensuring that image import is finished).

Thanks for Quick fix Edward,
Earlier I tried to give a fix but could not find a quick fix. Is it
possible to put some sleep says 5 minutes before we retry so that there are
chances that image will be imported. As you mentioned it was working before
and suddenly got stopped working without knowing the root caused.

I will definitely +1 for submitting the patch and i did it for power8
enablement and few other. But this one I could not thought of any solution.

Thanks and Regards,
Narinder Gupta (PMP) <email address hidden>
Canonical, Ltd. narindergupta [irc.freenode.net]
+1.281.736.5150 narindergupta2007[skype]

Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com

On Wed, Oct 5, 2016 at 1:27 PM, Edward Hope-Morley <
<email address hidden>> wrote:

> NOTE: to fix this we are basically skipping the images import check
> until we have a proper workaround. As a result, if commissioning is
> started before image are fully imported it might fail. The simple
> solution to this is to go into the UI and restart the commission
> (ensuring that image import is finished).
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1630325
>
> Title:
> maas-deployer failed ValueError: No JSON object could be decoded
>
> Status in maas-deployer:
> In Progress
> Status in OPNFV:
> New
>
> Bug description:
> maas-deployer failed with the error right after checking the status of
> the import boot images.
>
> 016-10-04 09:38:53,952 DEBUG Command executed successfully:
> stdout='Import of boot resources started'
> 2016-10-04 09:38:53,952 DEBUG Logging into 172.16.50.5
> Traceback (most recent call last):
> File "/usr/bin/maas-deployer", line 9, in <module>
> load_entry_point('maas-deployer==0.0.7', 'console_scripts',
> 'maas-deployer')()
> File "/usr/lib/python2.7/dist-packages/maas_deployer/cli.py", line
> 88, in main
> engine.deploy(target)
> File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/engine.py",
> line 77, in deploy
> self.wait_for_import_boot_images(client, maas_config)
> File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/engine.py",
> line 449, in wait_for_import_boot_images
> while not checker.did_downloads_start():
> File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/maasclient/bootimages.py",
> line 82, in did_downloads_start
> status = self.get_status()
> File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/maasclient/bootimages.py",
> line 78, in get_status
> data = json.loads(content)
> File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
> return _default_decoder.decode(s)
> File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
> raise ValueError("No JSON object could be decoded")
> ValueError: No JSON object could be decoded
>
>
> Here is the deployment.yaml and deployment logs.
> http://paste.ubuntu.com/2...

Hi Narinder, a sleep timeout will never work for everybody since downloads can take any amount of time. So for now just restart the commission in the UI if it fails, or just don't commission until you know that images are imported...maybe do a two-phase deploy. A better solution to the previous one would probably be to query the MAAS api to check what images are currently available, waiting for the expected ones to be there, and then proceed. Your welcome to submit a patch to do that.

The backport is now available in ppa:maas-deloyers/proposed

Edward,
I asked Bryan to try this and it looks like MAAS need more time before it
can create the nodes in MAAS. As with proposed he is getting this issue.

https://usercontent.irccloud-cdn.com/file/KEPn4Ea3/maas_deployer.log

where adding nodes are failing

Thanks and Regards,
Narinder Gupta (PMP) <email address hidden>
Canonical, Ltd. narindergupta [irc.freenode.net]
+1.281.736.5150 narindergupta2007[skype]

Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com

On Thu, Oct 6, 2016 at 5:20 AM, Edward Hope-Morley <
<email address hidden>> wrote:

> Hi Narinder, a sleep timeout will never work for everybody since
> downloads can take any amount of time. So for now just restart the
> commission in the UI if it fails, or just don't commission until you
> know that images are imported...maybe do a two-phase deploy. A better
> solution to the previous one would probably be to query the MAAS api to
> check what images are currently available, waiting for the expected ones
> to be there, and then proceed. Your welcome to submit a patch to do
> that.
>
> The backport is now available in ppa:maas-deloyers/proposed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1630325
>
> Title:
> maas-deployer failed ValueError: No JSON object could be decoded
>
> Status in maas-deployer:
> In Progress
> Status in OPNFV:
> New
>
> Bug description:
> maas-deployer failed with the error right after checking the status of
> the import boot images.
>
> 016-10-04 09:38:53,952 DEBUG Command executed successfully:
> stdout='Import of boot resources started'
> 2016-10-04 09:38:53,952 DEBUG Logging into 172.16.50.5
> Traceback (most recent call last):
> File "/usr/bin/maas-deployer", line 9, in <module>
> load_entry_point('maas-deployer==0.0.7', 'console_scripts',
> 'maas-deployer')()
> File "/usr/lib/python2.7/dist-packages/maas_deployer/cli.py", line
> 88, in main
> engine.deploy(target)
> File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/engine.py",
> line 77, in deploy
> self.wait_for_import_boot_images(client, maas_config)
> File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/engine.py",
> line 449, in wait_for_import_boot_images
> while not checker.did_downloads_start():
> File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/maasclient/bootimages.py",
> line 82, in did_downloads_start
> status = self.get_status()
> File "/usr/lib/python2.7/dist-packages/maas_deployer/vmaas/maasclient/bootimages.py",
> line 78, in get_status
> data = json.loads(content)
> File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
> return _default_decoder.decode(s)
> File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
> raise ValueError("No JSON object could be decoded")
> ValueError: No JSON object could be decoded
>
>
> Here is the deployment.yaml and...

FWIW, available imported images can be retrieved through API. If only one cluster controller *and* only one image to be imported, we could wait with a logic something like:

while [ "$(maas maas-profile boot-images read "$(maas maas-profile node-groups list | jq -r .[].uuid)")" = '[]' ];
    do sleep 5
done