Limit maximum time a build can take

Bug #890860 reported by Paul Sokolovsky
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Linaro Android Infrastructure
Fix Released
Medium
Georgy Redkozubov

Bug Description

Recently, I saw cases when builds took ~14hrs to complete (with failure). This happens due to poor scalability of concurrent builds and people not paying attention to already running builds and pumping bunch of them at the same time. Such builds usually hang in repo fetch for hours before being aborted due to git error, EC2 timeouts, or manual action. While improving scalability is hot and important topic, we need some stop-gap measures to deal with the current situation which become.

Proposed solution is to install Build timeout plugin for Jenkins https://wiki.jenkins-ci.org/display/JENKINS/Build-timeout+Plugin and cap max build time at, say, 2.5hrs which would still allow ~4 concurrent builds, but will kill doomed to fail builds in other cases not too late.

Related branches

Changed in linaro-android-infrastructure:
importance: Undecided → High
status: New → Triaged
assignee: nobody → Paul Sokolovsky (pfalcon)
Changed in linaro-android-infrastructure:
status: Triaged → In Progress
Revision history for this message
Paul Sokolovsky (pfalcon) wrote :

This actually wasn't started - switch to seeded builds work was made.

Changed in linaro-android-infrastructure:
status: In Progress → Triaged
importance: High → Medium
Revision history for this message
Paul Sokolovsky (pfalcon) wrote :

The issue largely resolved with seeded builds deployment, but still would be nice to have such protection.

Changed in linaro-android-infrastructure:
importance: Medium → Low
Changed in linaro-android-infrastructure:
importance: Low → Medium
Changed in linaro-android-infrastructure:
assignee: Paul Sokolovsky (pfalcon) → Georgy Redkozubov (gesha)
status: Triaged → In Progress
Changed in linaro-android-infrastructure:
milestone: none → 2012.02
status: In Progress → Fix Committed
Changed in linaro-android-infrastructure:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.