UTAH

utah fails after reboot, but test continues to run on SUT

Bug #1179531 reported by Chris Gagnon on 2013-05-13

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	UTAH	Fix Released	High	Andy Doan	UTAH 0.12.4

Bug Description

Description:
utah returns failure after testcase with reboot step, but test continues to run on SUT after utah has failed.

the console log can be found here:
http://10.97.2.10:8080/job/mir-glmark-daily-new/54/console

I've also attached the /var/lib/utah folder

Related branches

lp://staging/~doanac/utah/bug1179531

Merged into lp://staging/~utah/utah/0.12 at revision 898

Andy Doan (community): Approve on 2013-05-15

lp://staging/utah

Merged into lp://staging/~nuclearbob/utah/trunk at revision 265

Max Brustkern: Pending requested 2012-12-11

Diff: 10666 lines (+6316/-1397)

85 files modified

COPYING (+674/-0)
Makefile (+1/-1)
client-Makefile (+4/-1)
client.py (+15/-0)
conf/utah/uqt-vm-tools.conf (+1/-1)
debian/changelog (+101/-0)
debian/control (+17/-3)
debian/rules (+21/-2)
debian/utah.install (+2/-0)
debian/utah.postinst (+10/-2)
distribute_setup.py (+3/-1)
docs/source/conf.py (+42/-23)
docs/source/reference.rst (+17/-0)
examples/run_install_test.py (+43/-18)
examples/run_test_bamboo_feeder.py (+123/-0)
examples/run_test_cobbler.py (+55/-25)
examples/run_test_vm.py (+38/-14)
examples/run_utah_tests.py (+51/-31)
examples/utah_logs.sh (+3/-1)
examples/utah_syslog.sh (+1/-1)
setup.py (+20/-6)
tests/test_parser.py (+60/-0)
tests/test_preseed.py (+476/-33)
utah/__init__.py (+15/-0)
utah/client/__init__.py (+15/-0)
utah/client/common.py (+70/-32)
utah/client/examples/test_state_partial_all_failed.yaml (+90/-0)
utah/client/examples/test_state_partial_inprogress.yaml (+90/-0)
utah/client/examples/test_state_partial_run_all.yaml (+90/-0)
utah/client/exceptions.py (+15/-0)
utah/client/phoenix.py (+40/-25)
utah/client/result.py (+48/-22)
utah/client/runner.py (+66/-58)
utah/client/state_agent.py (+15/-0)
utah/client/testcase.py (+73/-51)
utah/client/tests/__init__.py (+14/-0)
utah/client/tests/common.py (+43/-0)
utah/client/tests/manual_privileges.py (+17/-2)
utah/client/tests/test_common.py (+22/-7)
utah/client/tests/test_jsonschema.py (+21/-49)
utah/client/tests/test_misc.py (+15/-0)
utah/client/tests/test_phoenix.py (+25/-10)
utah/client/tests/test_result.py (+22/-5)
utah/client/tests/test_runner.py (+22/-7)
utah/client/tests/test_state_agent.py (+134/-68)
utah/client/tests/test_testcase.py (+46/-31)
utah/client/tests/test_testsuite.py (+45/-23)
utah/client/tests/test_vcs_bzr.py (+15/-0)
utah/client/tests/test_vcs_dev.py (+15/-0)
utah/client/tests/test_vcs_git.py (+15/-0)
utah/client/tests/test_yaml.py (+51/-36)
utah/client/testsuite.py (+31/-13)
utah/commandstr.py (+15/-0)
utah/config.py (+192/-18)
utah/exceptions.py (+15/-12)
utah/group.py (+15/-0)
utah/iso.py (+277/-118)
utah/isotest/data/file_list_di (+23/-23)
utah/isotest/data/file_list_ubiquity (+21/-21)
utah/isotest/iso_static_validation.py (+85/-60)
utah/orderedcollections.py (+90/-0)
utah/parser.py (+211/-0)
utah/preseed.py (+669/-119)
utah/process.py (+20/-5)
utah/provisioning/__init__.py (+15/-0)
utah/provisioning/baremetal/__init__.py (+15/-0)
utah/provisioning/baremetal/bamboofeeder.py (+301/-0)
utah/provisioning/baremetal/cobbler.py (+124/-130)
utah/provisioning/baremetal/exceptions.py (+15/-0)
utah/provisioning/baremetal/power.py (+90/-0)
utah/provisioning/exceptions.py (+15/-0)
utah/provisioning/inventory/__init__.py (+15/-0)
utah/provisioning/inventory/exceptions.py (+15/-0)
utah/provisioning/inventory/inventory.py (+20/-5)
utah/provisioning/inventory/sqlite.py (+87/-52)
utah/provisioning/provisioning.py (+337/-107)
utah/provisioning/vm/__init__.py (+15/-0)
utah/provisioning/vm/exceptions.py (+15/-0)
utah/provisioning/vm/libvirtvm.py (+124/-70)
utah/provisioning/vm/vm.py (+16/-0)
utah/publish.py (+146/-0)
utah/retry.py (+15/-0)
utah/run.py (+142/-34)
utah/timeout.py (+21/-4)
utah/url.py (+182/-17)

lp://staging/~nuclearbob/utah/bug1179531

Merged into lp://staging/utah at revision 908

Javier Collado (community): Approve on 2013-05-17

Andy Doan (community): Approve on 2013-05-16

Revision history for this message

Chris Gagnon (chris.gagnon) wrote on 2013-05-13:

utah.tar.gz Edit (1.7 KiB, application/x-tar)

Javier Collado (javier.collado) on 2013-05-13

Changed in utah:
importance:	Undecided → High

Andy Doan (doanac) on 2013-05-13

Changed in utah:
status:	New → Confirmed

Revision history for this message

Andy Doan (doanac) wrote on 2013-05-13:

I think this is because our ssh.py code now throws a UTAHProvisioningException, but our is_utah_done logic is still just trying to catch socket errors. This causes the retry logic to fail. Still digging into this.

Revision history for this message

Andy Doan (doanac) wrote on 2013-05-13:

changing that exception seems to get things going. the log gets cluttered with messages of utah waiting for it to be done, but that's expected.

Andy Doan (doanac) on 2013-05-14

Changed in utah:
assignee:	nobody → Andy Doan (doanac)

Revision history for this message

Chris Gagnon (chris.gagnon) wrote on 2013-05-14:

I've worked around this issue with this hack in my jenkins job.

get_results () {
rsync -avz --remove-source-files -e 'ssh -i /var/lib/jenkins/.ssh/utah -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' jenkins@$machine:/tmp/results.json . || RETCODE=1
}

until get_results && cat -n results.json | grep "fragment-steps=0" | grep 'vertex-steps=0'; do
sleep 10
done

Andy Doan (doanac) on 2013-05-15