I deployed landscape 16.03 from the 16.03 series, ceph/swift, with the image sync feature flag enabled. It failed because no images were ever downloaded. The gs3 charm was deployed with no errors, but the cron job was backtracing:
ERROR * 04-01 23:09:03 [PID:21584] * root * Exception during do_sync
Traceback (most recent call last):
File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 541, in main
"message": "Sync starting."})
File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 467, in send_message
with self.conn.Producer(exchange=self.exchange) as producer:
File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 660, in Producer
return Producer(channel or self, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
self.revive(self._channel)
File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
self.declare()
File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
self.exchange.declare()
File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 163, in declare
return self.channel.exchange_declare(
File "/usr/lib/python2.7/dist-packages/kombu/abstract.py", line 115, in channel
channel = self._channel = channel()
File "/usr/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 405, in __call__
value = self.__value__ = self.__contract__()
File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 203, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 766, in default_channel
self.connection
File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 751, in connection
self._connection = self._establish_connection()
File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 710, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/lib/python2.7/dist-packages/kombu/transport/librabbitmq.py", line 119, in establish_connection
conn = self.Connection(**opts)
File "/usr/lib/python2.7/dist-packages/librabbitmq/__init__.py", line 184, in __init__
self.connect()
ConnectionError: Error opening socket: Bad file descriptor
Further debugging showed that it was trying to connect to a rabbitmq-host with the ip "1" (yes, just 1). Guess why:
in glance-simplestreams-sync.py, StatusExchange() class, _setup_connection() method:
(Pdb) p hosts
'10.245.201.15,10.245.201.253,10.245.201.34' <--- these are my rabbit hosts
(Pdb) l
429
430 id_conf = read_conf(ID_CONF_FILE_NAME)
431
432 import pdb; pdb.set_trace()
433 hosts = id_conf.get('rabbit_hosts', None)
434 -> if hosts is not None:
435 host = hosts[0]
436 else:
437 host = id_conf.get('rabbit_host', None)
438
439 if host is None:
(Pdb) type(hosts)
<type 'str'>
(Pdb)
It works for a single rabbit (or a simple 3 node ceph/ceph deployment) because config variables are different between single and multiple rabbit units.
When deployment has only one rabbit unit: config variable is "rabbit_host", singular, and glance_simplestreams_sync.py gets the variable as is, get("rabbit_host", None), which works.
When deployment has more than one rabbit unit: config variable is "rabbit_hosts", plural, and after retrieving the value it tries to get the first element of the hosts list, which is a char given hosts is a string. A split(",") is missing there.
It's a bug in the charm, but looks like we are using our own copy. Since we will need to change landscape to at least bump the charm revision, I'm filing the bug against landscape first.
It works for a single rabbit (or a simple 3 node ceph/ceph deployment) because codepath is different as config variables are different for single and multiple rabbit units.
When deployment has only one rabbit unit, config variable is called "rabbit_host", singular, and glance_ simplestreams_ sync.py gets the variable as is, get("rabbit_host", None), which works.
When deployment has more than one rabbit unit, config variable is called "rabbit_hosts", plural, and after retrieving the value it tries to get the first element of the hosts list, which is a char given hosts is a string. A split(",") is missing there.