Trying to enable statistics for one repository on the staging instance, leads to this error in the log:
2013-05-29 16:17:45.897 ERROR [rhodecode.lib.celerylib] Traceback (most recent call last):
File "/home/rhodecode/.local/lib/python2.7/site-packages/RhodeCode-1.5.4_linaro5-py2.7.egg/rhodecode/lib/celerylib/__init__.py", line 64, in run_task
t = task.apply_async(args=args, kwargs=kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery-2.2.10-py2.7.egg/celery/task/base.py", line 448, in apply_async
exchange_type=exchange_type)
File "/usr/local/lib/python2.7/dist-packages/celery-2.2.10-py2.7.egg/celery/task/base.py", line 301, in get_publisher
**options)
File "/usr/local/lib/python2.7/dist-packages/celery-2.2.10-py2.7.egg/celery/app/amqp.py", line 328, in TaskPublisher
return TaskPublisher(*args, **self.app.merge(defaults, kwargs))
File "/usr/local/lib/python2.7/dist-packages/celery-2.2.10-py2.7.egg/celery/app/amqp.py", line 156, in __init__
super(TaskPublisher, self).__init__(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/kombu-1.5.1-py2.7.egg/kombu/compat.py", line 57, in __init__
super(Publisher, self).__init__(connection, self.exchange, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/kombu-1.5.1-py2.7.egg/kombu/messaging.py", line 69, in __init__
channel = channel.default_channel
File "/usr/local/lib/python2.7/dist-packages/kombu-1.5.1-py2.7.egg/kombu/connection.py", line 585, in default_channel
self._default_channel = self.channel()
File "/usr/local/lib/python2.7/dist-packages/kombu-1.5.1-py2.7.egg/kombu/connection.py", line 171, in channel
chan = self.transport.create_channel(self.connection)
File "/usr/local/lib/python2.7/dist-packages/kombu-1.5.1-py2.7.egg/kombu/connection.py", line 578, in connection
self._connection = self._establish_connection()
File "/usr/local/lib/python2.7/dist-packages/kombu-1.5.1-py2.7.egg/kombu/connection.py", line 539, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python2.7/dist-packages/kombu-1.5.1-py2.7.egg/kombu/transport/pyamqplib.py", line 255, in establish_connection
connect_timeout=conninfo.connect_timeout)
File "/usr/local/lib/python2.7/dist-packages/kombu-1.5.1-py2.7.egg/kombu/transport/pyamqplib.py", line 52, in __init__
super(Connection, self).__init__(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 144, in __init__
(10, 30), # tune
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait
self.channel_id, allowed_methods)
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 202, in _wait_method
self.method_reader.read_method()
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
raise m
IOError: Socket closed
Locally, this is not happening. I think this is due to the way the machine was once set up. RhodeCode was once erroneously installed system wide via "pip". I guess this took with it some dependencies that are now causing the problem.
This morning I cleaned up the dependencies on the staging instance, but that did not solve the problem.
Taking a look at other logs (celeryd and rabbitmq), looks like RhodeCode was not able to connect anymore to the broker server. For some unknown reasons - I guess an installation or an upgrade not concluded correctly - the password set in the RhodeCode configuration file was not the correct one to connect to the broker.
I had to "purge" the rabbitmq-server package, reinstall it and re-configure the correct user and vhost with a new password.
Testing now that everything works correctly.