UTAH

UTAH doesn't honor jobtimeout

Bug #1169510 reported by Jean-Baptiste Lallement on 2013-04-16

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	UTAH	Fix Released	High	Javier Collado	UTAH 0.12

Bug Description

In job [1] jobtimeout is set to 7200s (2hours) but it is still running after more than 2.5hours

[1] http://10.97.0.1:8080/job/ps-generic-autopilot-release-testing/76/label=autopilot-nvidia/?

Tags:

Related branches

lp://staging/~javier.collado/utah/bug1169510

Merged into lp://staging/utah at revision 866

Max Brustkern (community): Approve on 2013-04-18

Andy Doan (community): Approve on 2013-04-18

Revision history for this message

Javier Collado (javier.collado) wrote on 2013-04-16:

I've downloaded the configuration file being used for the job and tested that
the same value is seen when looking at `config.jobtimeout`. Hence, the
configuration is correct, but indeed the timeout isn't working as expected.

Changed in utah:
status:	New → Confirmed
importance:	Undecided → High

Revision history for this message

Javier Collado (javier.collado) wrote on 2013-04-17:

Example script to reproduce the problem Edit (734 bytes, text/x-python)

After some testing, the behavior I've seen confirms the following python bug:
http://bugs.python.org/issue9504

What this bug says is that if a command if being executed through a subprocess,
the signal handler isn't called until the subprocess returns.

What happened in this specific instance of the problem is that the utah client
was being executed through ssh and the signal was already triggered, but the
handler wasn't called.

I've attached a sample script to reproduce the problem. When `signal.alarm` is
used, the handler isn't called until the ssh command is done. However if the
timeout is set using `channel.settimeout` a `socket.timeout` exeception is
raised at the expected time.

Javier Collado (javier.collado) on 2013-04-18