Cleanup tool needs some improvements

Bug #1546022 reported by Proskurin Kirill
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-mesos
New
Undecided
Unassigned

Bug Description

Sometimes cleanup catch some strange container, or it's a race or something:
INFO - Removing container /mesos-e3cdf4a5-37ba-44bb-b0e1-13e631fd30b8-S0.ae97745a-f4f0-4c18-ac7f-dd31e864129d
INFO - Removing container /mesos-e3cdf4a5-37ba-44bb-b0e1-13e631fd30b8-S0.f8833c22-a5bb-4954-9b39-c771d4a6be85
INFO - Removing container /mesos-e3cdf4a5-37ba-44bb-b0e1-13e631fd30b8-S0.44ceeb2d-de6c-4c06-bc0b-46a8f7265f78
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 380, in _handle_results
    task = get()
TypeError: ('__init__() takes at least 3 arguments (2 given)', <class 'docker.errors.NotFound'>, (HTTPError(u'404 Client Error: Not Found for url: /v1.21/containers/mesos-e3cdf4a5-37ba-44bb-b0e1-13e631fd30b8-S0.44ceeb2d-de6c-4c06-bc0b-46a8f7265f78?force=False&link=False&v=False',),))

And hangs.
At this point only mesos stack containers is running:
# docker ps -a | awk '{print $NF}'
NAMES
mesos_slave
chronos
marathon
mesos_master
zookeeper

If you try to push Ctrl+C you will get this:

^CProcess PoolWorker-2:
Process PoolWorker-3:
Process PoolWorker-1:
Process PoolWorker-5:
Process PoolWorker-4:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    self._target(*self._args, **self._kwargs)
    self.run()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
    task = get()
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in get
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in get
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in get
    racquire()
    racquire()
    return recv()
KeyboardInterrupt
    racquire()
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
    racquire()
KeyboardInterrupt

And it still will be hanged.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.