ppa-cleaner failed to start

Bug #1285386 reported by Francis Ginther
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Ubuntu CI Engine
Fix Released
High
Vincent Ladeuil
Ubuntu CI Services
Fix Released
High
Vincent Ladeuil

Bug Description

I see cases where the build ppas are not cleaned. The API was being called by the lander_ppa_free job calls: Calling via PATCH [http://10.55.32.150:8080/api/v1/ppa/1/]: {'state': 'dirty'}

The ppa-cleaner upstart log showed:
INFO:root:populating ppa model from: https://api.launchpad.net/1.0/~fginther/ppas
ESC[31;1mOperationalError: unable to open database file

Status of service:
$ curl http://10.55.32.150:8080/api/v1/ppa/{"meta": {"limit": 20, "next": null, "offset": 0, "previous": null, "total_count": 2}, "objects": [{"id": 1, "name": "ppa:fginther/ci-pool-001", "resource_uri": "/api/v1/ppa/1/", "state": 20, "ticket_id": 1, "timestamp": "2014-02-26T20:18:58.010577"}, {"id": 2, "name": "ppa:fginther/ci-pool-002", "resource_uri": "/api/v1/ppa/2/", "state": 20, "ticket_id": 1, "timestamp": "2014-02-26T20:44:01.642353"}]}

The upstart job was not running, after starting it with "sudo start ppa-cleaner", the status soon went to:
$ curl http://10.55.32.150:8080/api/v1/ppa/{"meta": {"limit": 20, "next": null, "offset": 0, "previous": null, "total_count": 2}, "objects": [{"id": 1, "name": "ppa:fginther/ci-pool-001", "resource_uri": "/api/v1/ppa/1/", "state": 30, "ticket_id": 1, "timestamp": "2014-02-26T21:48:25.595955"}, {"id": 2, "name": "ppa:fginther/ci-pool-002", "resource_uri": "/api/v1/ppa/2/", "state": 30, "ticket_id": 1, "timestamp": "2014-02-26T21:48:58.883521"}]}f

Tags: airline

Related branches

Revision history for this message
Andy Doan (doanac) wrote :

you can work around this right now, by doing

  juju ssh ppa-django/0 sudo restart ppa-cleaner

i need to do some charm debugging

Changed in ubuntu-ci-services-itself:
importance: Undecided → High
Andy Doan (doanac)
Changed in ubuntu-ci-services-itself:
milestone: none → phase-0
Revision history for this message
Andy Doan (doanac) wrote :

I'm going to try and test this patch locally:

 http://paste.ubuntu.com/7030056/

as I've never seen this error, I'm not sure that me saying this fixes it actually means anything.

Andy Doan (doanac)
Changed in ubuntu-ci-services-itself:
status: New → In Progress
Changed in ubuntu-ci-services-itself:
status: In Progress → Fix Committed
Andy Doan (doanac)
Changed in ubuntu-ci-services-itself:
status: Fix Committed → Fix Released
Revision history for this message
Andy Doan (doanac) wrote :

psivaa is seeing the issue again today even with the new patch :/

Changed in ubuntu-ci-services-itself:
status: Fix Released → Confirmed
Revision history for this message
Vincent Ladeuil (vila) wrote :

We had issues yesterday because the ppa cleaner was doing a too good job and it was hard to keep packages alive ;)

I ended up connecting to jenkins and commenting out the service_wrapper call in the ppa cleaner job definition.

Can this still be reproduced on trunk ?

Revision history for this message
Vincent Ladeuil (vila) wrote :

FTR: the jenkins job is at: http://<juju status lander-jenkins public-address>:8080/job/lander_ppa_free/configure

Revision history for this message
Andy Doan (doanac) wrote : Re: [Bug 1285386] Re: ppa-cleaner failed to start

On 03/13/2014 02:41 AM, Vincent Ladeuil wrote:
> Can this still be reproduced on trunk ?

yeah - psivaa saw it *after* I "fixed" this :/

Changed in ubuntu-ci-services-itself:
status: Confirmed → Fix Released
Changed in ubuntu-ci-services-itself:
status: Fix Released → Confirmed
Evan (ev)
Changed in ubuntu-ci-services-itself:
milestone: phase-0 → phase-0.5
Revision history for this message
Vincent Ladeuil (vila) wrote :

$ juju run --unit ppa-django/0 'sudo zcat /var/log/upstart/ppa-cleaner.log.1.gz'

INFO:root:populating ppa model from: https://api.launchpad.net/1.0/~vila/ppas
INFO:root:adding new ppas: set([])
INFO:root:removing ppas: set([])
INFO:root:populating ppa model from: https://api.launchpad.net/1.0/~vila/ppas
ERROR:root:Could not populate PPAs from launchpad
Traceback (most recent call last):
  File "/srv/ppa_django/ppa-assigner/ppa_assigner/models.py", line 92, in populate_from_launchpad
    for ppa in launchpad.lp_collection(url):
  File "/srv/ppa_django/ppa-assigner/ppa_assigner/launchpad.py", line 27, in lp_collection
    resp = _lp_get_authenticated(url)
  File "/srv/ppa_django/ppa-assigner/ppa_assigner/launchpad.py", line 70, in _lp_get_authenticated
    return urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 502: Bad Gateway
HTTPError: HTTP Error 502: Bad Gateway

While this can be caught and retried (known transient lp issue), the job itself should at least respawn itself.

Changed in ubuntu-ci-services-itself:
status: Confirmed → In Progress
assignee: Andy Doan (doanac) → Vincent Ladeuil (vila)
Changed in ubuntu-ci-services-itself:
status: In Progress → Fix Committed
Vincent Ladeuil (vila)
Changed in ubuntu-ci-services-itself:
status: Fix Committed → Fix Released
Ursula Junque (ursinha)
Changed in uci-engine:
assignee: nobody → Vincent Ladeuil (vila)
importance: Undecided → High
milestone: none → phase-0.5
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.