Christian Robottom Reis wrote:
> Interesting mirror prober error output:
> 
> ----- Forwarded message from Cron Daemon <email address hidden> -----
> 
> From: Cron Daemon <email address hidden>
> To: <email address hidden>
> Subject: Cron <launchpad@gangotri> /srv/launchpad.net/nightly.sh
> Date: Mon, 14 Aug 2006 12:25:15 +0100 (BST)
> Reply-To: <email address hidden>
> X-Spambayes-Classification: ham; 0.00
> 
> == Distribution mirror prober (release) Sun Aug 13 06:41:32 UTC 2006 ==
> 06:41:33 INFO    Probing Release Mirrors
> 06:41:35 WARNING Mirror 'primary-baltix-download' of distribution 'baltix' can't be probed --we only probe Ubuntu mirrors.
> 06:41:35 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:36 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:36 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:37 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:37 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:38 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:38 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:39 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:40 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:40 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:41 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:41 WARNING Mirror 'ftp.rediris.es-release' of distribution 'ubuntu' doesn't have an http base URL, we can't probe it.
> 06:41:41 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 06:41:42 ERROR   Unable to fetch http://releases.ubuntu.com/.manifest: HTTP Error 503: Service Unavailable
> 11:15:32 INFO    Probed 13 mirrors.
> Traceback (most recent call last):
>   File "distributionmirror-prober.py", line 240, in ?
>     sys.exit(main(sys.argv))
>   File "distributionmirror-prober.py", line 224, in main
>     _create_probe_record(mirror, logfiles[mirror.id])
>   File "distributionmirror-prober.py", line 155, in _create_probe_record
>     file=logfile, contentType='text/plain')
>   File "/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/database/librarian.py", line 94, in create
>     fid = client.addFile(name, size, file, contentType, expires, debugID)
>   File "/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/librarian/client.py", line 78, in addFile
>     raise UploadFailed('Invalid length: %d' % size)
> canonical.librarian.interfaces.UploadFailed: Invalid length: 0
> 
> ----- End forwarded message -----
> 
> At least the latter seems like a bug -- but why would the filesize be
> zero?

	affects /products/launchpad
	status confirmed

More interesting, this was sent only after I killed this process as it had
been running for waay too long (note the 4.5 hours between he last ERROR and
the following INFO). And I just had to do it again today too so it was no a
one off problem.

Running it from the command line I get this:

launchpad@gangotri:/srv/launchpad.net/production/launchpad/cronscripts$
python distributionmirror-prober.py --content-type=archive
05:20:16 INFO    Probing Archive Mirrors
05:20:18 INFO    No mirrors to probe.
05:20:18 INFO    Done.
launchpad@gangotri:/srv/launchpad.net/production/launchpad/cronscripts$
python distributionmirror-prober.py --content-type=archive --force
05:20:28 INFO    Probing Archive Mirrors
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:
Traceback (most recent call last):
  File "distributionmirror-prober.py", line 240, in ?
    sys.exit(main(sys.argv))
  File "distributionmirror-prober.py", line 212, in main
    reactor.run()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 218, in run
    self.mainLoop()
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/posixbase.py",
line 226, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/base.py",
line 555, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 106, in failWithTimeoutError
    self.failed(ProberTimeout(self.url, self.timeout))
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/canonical/launchpad/scripts/distributionmirror_prober.py",
line 116, in failed
    self.deferred.errback(reason)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 251, in errback
    self._startRunCallbacks(fail)
  File
"/srv/launchpad.net/production/launchpad/cronscripts/../lib/twisted/internet/defer.py",
line 280, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError:

-- 
Stuart Bishop <email address hidden>   http://www.canonical.com/
Canonical Ltd.                                http://www.ubuntu.com/