Failure to create a Jenkins job should block a dependency creation on Capomastro
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Capomastro |
Triaged
|
High
|
Unassigned |
Bug Description
Today on Canonistack we had a disk filled up which started to cause issues because dependencies can be set up normally on Capomastro even though they are failing to be created on Jenkins (the unit with the full disk that stopped to respond normally).
This is what we had on Capomastro:
[2015-02-12 13:54:28,097: DEBUG/Worker-1] "POST /createItem?
Traceback (most recent call last):
File "/usr/lib/
R = retval = fun(*args, **kwargs)
File "/usr/lib/
return self.run(*args, **kwargs)
File "/usr/lib/
client.
File "/usr/lib/
return self.jobs.
File "/usr/lib/
params=params
File "/usr/lib/
return self.post_
File "/usr/lib/
response.url, data, headers, response.
JenkinsAPIExcep
Still the dependency was created okay on Capomastro, so the users tried to build them (although they were not found on Jenkins):
[2015-02-12 13:56:19,437: INFO/Worker-1] Starting new HTTP connection (1): 10.55.32.4
[2015-02-12 13:56:19,439: DEBUG/Worker-1] Setting read timeout to None
[2015-02-12 13:56:19,444: DEBUG/Worker-1] "GET /api/python HTTP/1.1" 200 437
[2015-02-12 13:56:19,458: ERROR/MainProcess] Task jenkins.
Traceback (most recent call last):
File "/usr/lib/
R = retval = fun(*args, **kwargs)
File "/usr/lib/
return self.run(*args, **kwargs)
File "/usr/lib/
client.
File "/usr/lib/
self[
File "/usr/lib/
raise UnknownJob(jobname)
UnknownJob: u'android-
When that happens no message is displayed to warn the user. Ideally the dependency creation IMHO should not even be allowed in that case, but probably the Jenkins API exception is not being properly handled.
Changed in capomastro: | |
milestone: | 2015-05 → june-2015 |
So I guess capomastro writes the record in the database, then fires off the task to create the job and forgets about it. I wonder if dependencies need to have a "jenkins-created" flag or field that's set to False and only set to True when the celery task succeeds. We should then disallow building dependencies not present in jenkins. We may also need a way to show alerts to the user so this is more in-your-face than just a tiny field in the dependencies table/view.
So it's a bit of work here, but is quite important to implement.