Node failed at deploy. Exception: int() argument must be a string or a number, not 'tuple'

This trace is ugly. Better version:

File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/agent_base_vendor.py", line 482, in heartbeat
  self.continue_deploy(task)
File "/usr/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
  result = f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/ironic/conductor/task_manager.py", line 138, in wrapper
  return f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py", line 382, in continue_deploy
  uuid_dict_returned = do_agent_iscsi_deploy(task, self._client)
File "/usr/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
  result = f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py", line 314, in do_agent_iscsi_deploy
  uuid_dict_returned = continue_deploy(task, iqn=iqn, address=address)
File "/usr/lib/python2.7/dist-packages/ironic_lib/metrics.py", line 61, in wrapped
  result = f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/iscsi_deploy.py", line 243, in continue_deploy
  uuid_dict_returned = deploy_utils.deploy_disk_image(**params)
File "/usr/lib/python2.7/dist-packages/ironic/drivers/modules/deploy_utils.py", line 388, in deploy_disk_image
  configdrive)
File "/usr/lib/python2.7/dist-packages/ironic_lib/disk_utils.py", line 722, in create_config_drive_partition
  if _is_disk_larger_than_max_size(device, node_uuid):
File "/usr/lib/python2.7/dist-packages/ironic_lib/disk_utils.py", line 571, in _is_disk_larger_than_max_size
  disksize_mb = int(disksize_bytes) // 1024 // 1024

Update: I set up instance creation with --config-drive=True option.

Thanks for the report George!

Can you provide some more information? Some useful starting information would be:

What OS/distro are you using?

How was ironic set up? (installation guide, devstack, packstack?)

Thanks!

Hello.

I done this with Xenial/newton distribution, but I think this bug should appears on any installation of Ironic/Newton.

Conditions:

0. integration with nova/neutron.
1. pxe_drac driver is in use for node.
2. instance is booted with --config-drive True

As far as I can see by the trace, issue happens somewhere near the code which writes config drive into node in pxe (iscsi) mode.

I'm not sure, but may be it can be reproduced in other pxe_* drivers (they should use iscsi to mount node drive to ironic-conductor node).

This looks like a bug in ironic-lib as it has to do with handling command output. That being said, the call could still be failing with the bug in the command output being fixed. You may wish to check your kernel log to see if you see any errors there. In the mean time, I'm working on a patch for the basic issue in ironic-lib.

Fix proposed to branch: master
Review: https://review.openstack.org/450754

Please give https://review.openstack.org/450754 a try in your ironic-lib, and see if the clears this issue up for you.

There is no related dmesg errors, only messages from iscsi:

[2752572.185425] sd 4:0:0:1: [sda] Synchronizing SCSI cache
[2754877.255269] scsi host5: iSCSI Initiator over TCP/IP
[2754877.517408] scsi 5:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
[2754877.518327] scsi 5:0:0:0: Attached scsi generic sg0 type 12
[2754877.519606] scsi 5:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
[2754877.520364] sd 5:0:0:1: Attached scsi generic sg1 type 0
[2754877.520956] sd 5:0:0:1: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[2754877.521549] sd 5:0:0:1: [sda] Write Protect is off
[2754877.521556] sd 5:0:0:1: [sda] Mode Sense: 69 00 00 08
[2754877.521864] sd 5:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[2754877.527445] sd 5:0:0:1: [sda] Attached SCSI disk
[2754897.817887] sda: sda1
[2754898.789711] sda: sda1
[2754901.652210] sd 5:0:0:1: [sda] Synchronizing SCSI cache
[2756295.626738] scsi host6: iSCSI Initiator over TCP/IP
[2756295.886870] scsi 6:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
[2756295.887637] scsi 6:0:0:0: Attached scsi generic sg0 type 12
[2756295.888337] scsi 6:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
[2756295.889156] sd 6:0:0:1: Attached scsi generic sg1 type 0
[2756295.892813] sd 6:0:0:1: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[2756295.893590] sd 6:0:0:1: [sda] Write Protect is off
[2756295.893608] sd 6:0:0:1: [sda] Mode Sense: 69 00 00 08
[2756295.893944] sd 6:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[2756295.899332] sd 6:0:0:1: [sda] Attached SCSI disk
[2756316.290751] sda: sda1
[2756317.296678] sda: sda1
[2756320.197870] sd 6:0:0:1: [sda] Synchronizing SCSI cache
[2757572.317804] scsi host7: iSCSI Initiator over TCP/IP
[2757572.576966] scsi 7:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
[2757572.577919] scsi 7:0:0:0: Attached scsi generic sg0 type 12
[2757572.580114] scsi 7:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
[2757572.580889] sd 7:0:0:1: Attached scsi generic sg1 type 0
[2757572.582723] sd 7:0:0:1: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[2757572.585768] sd 7:0:0:1: [sda] Write Protect is off
[2757572.585773] sd 7:0:0:1: [sda] Mode Sense: 69 00 00 08
[2757572.588952] sd 7:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[2757572.594710] sd 7:0:0:1: [sda] Attached SCSI disk
[2757592.882001] sda: sda1

From the provided trace I'm wondered what version of ironic-lib you are using? In newton the code looks like [1]

File "/usr/lib/python2.7/dist-packages/ironic_lib/disk_utils.py", line 571, in _is_disk_larger_than_max_size
  disksize_mb = int(disksize_bytes) // 1024 // 1024

[1] https://github.com/openstack/ironic-lib/blob/stable/newton/ironic_lib/disk_utils.py#L599
disksize_mb = int(disksize_bytes[0].strip()) // 1024 // 1024

My version:

    disksize_mb = int(disksize_bytes) // 1024 // 1024
    return disksize_mb > MAX_DISK_SIZE_MB_SUPPORTED_BY_MBR

ironic-lib version: python-ironic-lib 2.1.0-0ubuntu1~cloud0

Looks like you should ideally have 2.1.3, which is where this issue should have been fixed, and would have been the latest tag for newton deployments.

https://github.com/openstack/ironic-lib/blob/2.1.3/ironic_lib/disk_utils.py#L599

so indeed in version 2.1.0 there is a buggy code that was fixed by:
https://github.com/openstack/ironic-lib/commit/33af42c96556ec66087954f115697d79967efe91
and it was fixed in ironic-lib 2.2.0

Hello Ubuntu guys. Can you please upload proper version of ironic-lib into cloud-archive?

Relates to newton UCA.

looks like this has been fixed :)