launchpad-buildd

Launchpad Buildd doesn't clean up old respective build dir

Bug #32236 reported by Celso Providelo
22
This bug affects 1 person
Affects Status Importance Assigned to Milestone
launchpad-buildd
Triaged
High
Unassigned

Bug Description

Some odd occurs in launchpad-buildd when it leaves a broken build directory (it's already a bug , it should be removed anyway). But when requested to build the same package it crashes as:

2006/02/21 04:31 GMT [HTTPChannel,452,82.211.81.167] Traceback (most recent call last):
          File "/usr/lib/python2.4/site-packages/twisted/protocols/http.py", line 557, in requestReceived
            self.process()
          File "/usr/lib/python2.4/site-packages/twisted/web/server.py", line 165, in process
            self.render(resrc)
          File "/usr/lib/python2.4/site-packages/twisted/web/server.py", line 172, in render
            body = resrc.render(self)
          File "/usr/lib/python2.4/site-packages/twisted/web/xmlrpc.py", line 130, in render
            defer.maybeDeferred(function, *args).addErrback(
        --- <exception caught here> ---
          File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 129, in maybeDeferred
            result = f(*args, **kw)
          File "/usr/share/launchpad-buildd/canonical/buildd/slave.py", line 484, in xmlrpc_build
            self.slave.manager.initiate(filemap, chrootsum, args)
          File "/usr/share/launchpad-buildd/canonical/buildd/debian.py", line 70, in initiate
            BuildManager.initiate(self, files, chroot, extra_args)
          File "/usr/share/launchpad-buildd/canonical/buildd/slave.py", line 115, in initiate
            os.mkdir("%s/build-%s" % (os.environ["HOME"], self._buildid))
        exceptions.OSError: [Errno 17] File exists: '/home/buildd/build-170168-4809'

Revision history for this message
Celso Providelo (cprov) wrote :

It blocks the bulder in a very unusual way, people take a lot of time to figure out it's broken.

Changed in launchpad-buildd:
assignee: nobody → cprov
status: Unconfirmed → Confirmed
Revision history for this message
Adam Conrad (adconrad) wrote :

I'll be adding a cleanup method, and tying it into startup and the abort methods to make sure this doesn't happen anymore.

Changed in launchpad-buildd:
assignee: cprov → adconrad
Revision history for this message
Celso Providelo (cprov) wrote :

Great, thanks.

Don't forget to send me the patch, since it's still maintained as RF code.

Revision history for this message
Celso Providelo (cprov) wrote :

I believe this issue is already fixed in production.

Changed in launchpad-buildd:
status: Triaged → Fix Released
Revision history for this message
Adam Conrad (adconrad) wrote :

This is definitely not fixed in production.

Changed in launchpad-buildd:
status: Fix Released → Confirmed
Adam Conrad (adconrad)
Changed in launchpad-buildd:
assignee: Adam Conrad (adconrad) → nobody
Revision history for this message
Robert Collins (lifeless) wrote :

Does this happen often?

Revision history for this message
Julian Edwards (julian-edwards) wrote :

I have never seen this happen.

Revision history for this message
William Grant (wgrant) wrote :

lamont complains about it often enough, so it does happen.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Doesn't the slave.clean() method clear up the crap?

Revision history for this message
William Grant (wgrant) wrote :

Yes, when it's called. But it isn't called when the builder dies or the master gets really confused.

Revision history for this message
Julian Edwards (julian-edwards) wrote : Re: [Bug 32236] Re: Launchpad Buildd doesn't clean up old respective build dir

On Friday 19 November 2010 11:31:22 you wrote:
> Yes, when it's called. But it isn't called when the builder dies or the
> master gets really confused.

Ok, we need to get rid of all that logic in the master and make the slave do
it when we start a new build.

This will need a bit of a protocol change as we currently do:

 * send file
 * send file
 ....
 * startbuild

Alternatively we can remove the slave clean call from the behavioral methods
and just do it once at the start of the build, equivalent to resuming a
virtual host.

Revision history for this message
Robert Collins (lifeless) wrote :

Doing it just in time before a build makes a lot of sense to me

On 20/11/2010 1:11 AM, "Julian Edwards" <email address hidden> wrote:

On Friday 19 November 2010 11:31:22 you wrote:
> Yes, when it's called. But it isn't called when the...
 Ok, we need to get rid of all that logic in the master and make the slave do
it when we start a new build.

This will need a bit of a protocol change as we currently do:

 * send file
 * send file
 ....
 * startbuild

Alternatively we can remove the slave clean call from the behavioral methods
and just do it once at the start of the build, equivalent to resuming a
virtual host.

--
Launchpad Buildd doesn't clean up old respective build dir
https://bugs.launchpad.net/bugs/3223...

Robert Collins (lifeless)
Changed in launchpad-buildd:
status: Confirmed → Triaged
Revision history for this message
LaMont Jones (lamont) wrote :

The most common occurances are:

1) launchpad-buildd gets upgraded to a new version, in the middle of a build.
2) the machine crashes mid-build (most notably armel)

#1 is a function of the development cycle on launchpad-buildd, #2 is just a function of builds overloading the poor hardware.

Revision history for this message
Robert Collins (lifeless) wrote :

We upgrade mid-builds? Cool. Scary. Cool.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.