DIB fails to build images, UnboundLocalError: local variable 'call_order' referenced before assignment

Bug #1718753 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Ian Wienand

Bug Description

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike/5a7120f/undercloud/home/jenkins/overcloud_image_build.log.txt.gz#_2017-09-21_16_45_14

2017-09-21 16:45:09 | Calculating image size (this may take a minute)...
2017-09-21 16:45:12 | 2017-09-21 16:45:12.871 INFO diskimage_builder.block_device.blockdevice [-] Wrote final block device config to [/tmp/dib_build.YHVbNXat/states/block-device/config.json]
2017-09-21 16:45:14 | 2017-09-21 16:45:14.267 INFO diskimage_builder.block_device.blockdevice [-] create() called
2017-09-21 16:45:14 | 2017-09-21 16:45:14.268 ERROR diskimage_builder.block_device.blockdevice [-] Create failed; rollback initiated
2017-09-21 16:45:14 | Traceback (most recent call last):
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/blockdevice.py", line 383, in cmd_create
2017-09-21 16:45:14 | dg, call_order = create_graph(self.config, self.params, state)
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/config.py", line 193, in create_graph
2017-09-21 16:45:14 | if node.name in dg.nodes:
2017-09-21 16:45:14 | TypeError: argument of type 'instancemethod' is not iterable
2017-09-21 16:45:14 | Traceback (most recent call last):
2017-09-21 16:45:14 | File "/usr/bin/dib-block-device", line 10, in <module>
2017-09-21 16:45:14 | sys.exit(main())
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/cmd.py", line 120, in main
2017-09-21 16:45:14 | return bdc.main()
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/cmd.py", line 115, in main
2017-09-21 16:45:14 | self.args.func()
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/cmd.py", line 36, in cmd_create
2017-09-21 16:45:14 | self.bd.cmd_create()
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/blockdevice.py", line 388, in cmd_create
2017-09-21 16:45:14 | reverse_order = reversed(call_order)
2017-09-21 16:45:14 | UnboundLocalError: local variable 'call_order' referenced before assignment
2017-09-21 16:45:15 | 2017-09-21 16:45:15.681 INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here
2017-09-21 16:45:15 | *** /tmp/dib_build.YHVbNXat/mnt is not a directory

Probably caused by:

https://github.com/openstack/diskimage-builder/commit/a47ff0dd4abbceae786d9a5a4d1ce8042a9eb4f7

Revision history for this message
Ben Nemec (bnemec) wrote :

This isn't just a promotion blocker, it's also going to break all jobs that build images. See http://logs.openstack.org/88/504588/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/8d35ffb/logs/undercloud/home/jenkins/overcloud_image_build.log.txt.gz

Although it looks like we have a serious mirror problem that's breaking things before we even get to that point now. Yay?

tags: added: alert
Revision history for this message
Ben Nemec (bnemec) wrote :

Proposed a revert at https://review.openstack.org/#/c/506376, although if that can't pass the gate because of networkx 2.0 then we're still out of luck. Worth a shot though.

Revision history for this message
Ian Wienand (iwienand) wrote :

Sorry, it looks like we're working in parallel a bit here.

There has been some confusion because we have put in a global pin of networkx<2.0, but *also* merged our 2.0 fix

I have put in https://review.openstack.org/506398 which reverts the change AND takes the networkx pin

In the changelog there I have outlined the plan. We have other changes in flight to make our testing correctly use constraints support. This way, we can restore the 2.0 support with a depends-on that bumps the requirements and test it properly. when other projects have updated in a similar way, we can do a global networkx bump in a controlled fashion

Revision history for this message
Ian Wienand (iwienand) wrote :

This should be fixed, and 2.9.0 is released with the pin.

constraints support is in [1], and a corresponding project-config change for the functional tests is in [2]

we can roll this out in a more controlled manner with [3] after those are in

[1] https://review.openstack.org/412747
[2] https://review.openstack.org/506388
[3] https://review.openstack.org/506524

Revision history for this message
wes hayutin (weshayutin) wrote :

looks resolved to me, thanks Ian!

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.