This is still about half hunch, but the theory is that in some cases Zuul will not reattach a change that has had its builds cancelled to the nearest non failing item when a change ahead of it has failed.
This appears to have happened to https://review.openstack.org/#/c/54221/. The jobs that show CANCELLED were cancelled because a change ahead of 54221 had failed. Zuul should then reattach 54221 to the nearest non failing item and restart the jobs but this didn't happen.
Logs that seem to support this:
2013-10-31 18:20:06,441 DEBUG zuul.DependentPipelineManager: Checking for changes needed by <Change 0x7fc00017e190 54221,1>:
2013-10-31 18:20:06,442 DEBUG zuul.DependentPipelineManager: Cancel jobs for change <Change 0x7fc00017e190 54221,1>
2013-10-31 18:20:06,442 DEBUG zuul.DependentPipelineManager: Found build <Build 35fa651eb8ec462f854e191cfe5b5ecd of gate-tempest-devstack-vm-full> for change <Change 0x7fc00017e190 54221,1> to cancel
2013-10-31 18:20:06,445 DEBUG zuul.DependentPipelineManager: Found build <Build b3c9074f0a044a6b89f90c12a284276f of gate-tempest-devstack-vm-postgres-full> for change <Change 0x7fc00017e190 54221,1> to cancel
2013-10-31 18:20:06,453 DEBUG zuul.DependentPipelineManager: Canceling jobs for change <Change 0x7fc0008413d0 54663,2>, behind change <Change 0x7fc00017e190 54221,1>
2013-10-31 18:20:07,442 DEBUG zuul.DependentPipelineManager: <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate> is a failing item because ['a needed change is failing', 'at least one job failed']
2013-10-31 18:20:07,442 INFO zuul.DependentPipelineManager: Resetting builds for change <Change 0x7fc0008413d0 54663,2> because the item ahead, <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate>, is not the nearest non-
failing item, <QueueItem 0x7fc0088a2290 for <Change 0x7fc000864690 50643,2> in gate>
2013-10-31 18:20:12,942 INFO zuul.DependentPipelineManager: Resetting builds for change <Change 0x7fc000de0f10 52639,1> because the item ahead, <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate>, is not the nearest non-failing item, <QueueItem 0x7fc000c9be50 for <Change 0x7fc0008413d0 54663,2> in gate>
2013-10-31 18:20:20,732 INFO zuul.DependentPipelineManager: Resetting builds for change <Change 0x7fc008482150 54444,2> because the item ahead, <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate>, is not the nearest non-failing item, <QueueItem 0x7fc000919e10 for <Change 0x7fc000de0f10 52639,1> in gate>
2013-10-31 18:20:23,158 INFO zuul.DependentPipelineManager: Resetting builds for change <Change 0x7fc00051c190 52988,9> because the item ahead, <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate>, is not the nearest non-failing item, <QueueItem 0x7fc0082c4310 for <Change 0x7fc008482150 54444,2> in gate>
2013-10-31 18:20:25,322 INFO zuul.DependentPipelineManager: Resetting builds for change <Change 0x7fc0081b5250 53182,9> because the item ahead, <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate>, is not the nearest non-failing item, <QueueItem 0x7fc009715a50 for <Change 0x7fc00051c190 52988,9> in gate>
2013-10-31 18:20:27,573 INFO zuul.DependentPipelineManager: Resetting builds for change <Change 0x7fc008832650 53869,5> because the item ahead, <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate>, is not the nearest non-failing item, <QueueItem 0x7fc00872ef90 for <Change 0x7fc0081b5250 53182,9> in gate>
2013-10-31 18:20:31,235 INFO zuul.DependentPipelineManager: Resetting builds for change <Change 0x7fc0004e8b10 53881,6> because the item ahead, <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate>, is not the nearest non-failing item, <QueueItem 0x7fc000518150 for <Change 0x7fc008832650 53869,5> in gate>
2013-10-31 18:20:34,973 INFO zuul.DependentPipelineManager: Resetting builds for change <Change 0x7fc00884b9d0 54659,4> because the item ahead, <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate>, is not the nearest non-failing item, <QueueItem 0x7fc000430e10 for <Change 0x7fc0004e8b10 53881,6> in gate>
2013-10-31 18:20:38,567 INFO zuul.DependentPipelineManager: Resetting builds for change <Change 0x7fc000833690 51688,11> because the item ahead, <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate>, is not the nearest non-failing item, <QueueItem 0x7fc000209a90 for <Change 0x7fc00884b9d0 54659,4> in gate>
Project openstack/tempest change 54221,1 based on <QueueItem 0x15a3f90 for <Change 0x7fc0088a28d0 54178,2> in gate>
2013-10-31 18:20:48,274 DEBUG zuul.DependentPipelineManager: Checking for changes needed by <Change 0x7fc00017e190 54221,1>:
2013-10-31 18:20:48,275 DEBUG zuul.DependentPipelineManager: Cancel jobs for change <Change 0x7fc00017e190 54221,1>
2013-10-31 18:20:48,284 DEBUG zuul.DependentPipelineManager: <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in gate> is a failing item because ['a needed change is failing', 'at least one job failed']
2013-10-31 18:20:50,250 DEBUG zuul.DependentPipelineManager: Re-enqueing change <Change 0x7fc00017e190 54221,1> in queue <ChangeQueue gate: openstack-dev/devstack, openstack-dev/grenade, openstack-dev/pbr, openstack-infra/devstack-gate, openstack-infra/jeepyb, openstack/ceilometer, openstack/cinder, openstack/glance, openstack/heat, openstack/horizon, openstack/keystone, openstack/neutron, openstack/nova, openstack/oslo.config, openstack/oslo.messaging, openstack/oslo.version, openstack/python-ceilometerclient, openstack/python-cinderclient, openstack/python-glanceclient, openstack/python-heatclient, openstack/python-keystoneclient, openstack/python-neutronclient, openstack/python-novaclient, openstack/requirements, openstack/swift, openstack/tempest, z/tempest>
2013-10-31 18:20:54,181 DEBUG zuul.DependentPipelineManager: Checking for changes needed by <Change 0x7fc00017e190 54221,1>:
2013-10-31 18:20:54,183 DEBUG zuul.DependentPipelineManager: <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in None> is a failing item because ['at least one job failed']
2013-10-31 18:20:54,183 INFO zuul.DependentPipelineManager: Resetting builds for change <Change 0x7fc0008413d0 54663,2> because the item ahead, <QueueItem 0x7fc000cb5710 for <Change 0x7fc00017e190 54221,1> in None>, is not the nearest non-failing item, <QueueItem 0x15a3f90 for <Change 0x7fc0088a28d0 54178,2> in None>
