glance-simplestreams-sync doesn't work with 3 rabbits

Bug #1565149 reported by Andreas Hasenack
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Landscape Server
Fix Released
High
Geoff Teale
glance-simplestreams-sync (Juju Charms Collection)
Fix Released
High
Geoff Teale

Bug Description

I deployed landscape 16.03 from the 16.03 series, ceph/swift, with the image sync feature flag enabled. It failed because no images were ever downloaded. The gs3 charm was deployed with no errors, but the cron job was backtracing:

ERROR * 04-01 23:09:03 [PID:21584] * root * Exception during do_sync
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 541, in main
    "message": "Sync starting."})
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 467, in send_message
    with self.conn.Producer(exchange=self.exchange) as producer:
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 660, in Producer
    return Producer(channel or self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
    self.revive(self._channel)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
    self.declare()
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
    self.exchange.declare()
  File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 163, in declare
    return self.channel.exchange_declare(
  File "/usr/lib/python2.7/dist-packages/kombu/abstract.py", line 115, in channel
    channel = self._channel = channel()
  File "/usr/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 405, in __call__
    value = self.__value__ = self.__contract__()
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 203, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 766, in default_channel
    self.connection
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 751, in connection
    self._connection = self._establish_connection()
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 710, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/lib/python2.7/dist-packages/kombu/transport/librabbitmq.py", line 119, in establish_connection
    conn = self.Connection(**opts)
  File "/usr/lib/python2.7/dist-packages/librabbitmq/__init__.py", line 184, in __init__
    self.connect()
ConnectionError: Error opening socket: Bad file descriptor

Further debugging showed that it was trying to connect to a rabbitmq-host with the ip "1" (yes, just 1). Guess why:

in glance-simplestreams-sync.py, StatusExchange() class, _setup_connection() method:
(Pdb) p hosts
'10.245.201.15,10.245.201.253,10.245.201.34' <--- these are my rabbit hosts
(Pdb) l
429
430 id_conf = read_conf(ID_CONF_FILE_NAME)
431
432 import pdb; pdb.set_trace()
433 hosts = id_conf.get('rabbit_hosts', None)
434 -> if hosts is not None:
435 host = hosts[0]
436 else:
437 host = id_conf.get('rabbit_host', None)
438
439 if host is None:
(Pdb) type(hosts)
<type 'str'>
(Pdb)

It works for a single rabbit (or a simple 3 node ceph/ceph deployment) because config variables are different between single and multiple rabbit units.
When deployment has only one rabbit unit: config variable is "rabbit_host", singular, and glance_simplestreams_sync.py gets the variable as is, get("rabbit_host", None), which works.
When deployment has more than one rabbit unit: config variable is "rabbit_hosts", plural, and after retrieving the value it tries to get the first element of the hosts list, which is a char given hosts is a string. A split(",") is missing there.

It's a bug in the charm, but looks like we are using our own copy. Since we will need to change landscape to at least bump the charm revision, I'm filing the bug against landscape first.

description: updated
tags: removed: kanban
Geoff Teale (tealeg)
Changed in landscape:
assignee: nobody → Geoff Teale (tealeg)
Geoff Teale (tealeg)
Changed in landscape:
status: New → In Progress
David Britton (dpb)
no longer affects: landscape/16.03
information type: Proprietary → Public
tags: added: kanban-cross-team landscape
Revision history for this message
Ursula Junque (ursinha) wrote :

It works for a single rabbit (or a simple 3 node ceph/ceph deployment) because codepath is different as config variables are different for single and multiple rabbit units.

When deployment has only one rabbit unit, config variable is called "rabbit_host", singular, and glance_simplestreams_sync.py gets the variable as is, get("rabbit_host", None), which works.
When deployment has more than one rabbit unit, config variable is called "rabbit_hosts", plural, and after retrieving the value it tries to get the first element of the hosts list, which is a char given hosts is a string. A split(",") is missing there.

description: updated
Changed in glance-simplestreams-sync (Juju Charms Collection):
status: New → Fix Committed
Changed in landscape:
status: In Progress → Fix Committed
Geoff Teale (tealeg)
Changed in glance-simplestreams-sync (Juju Charms Collection):
status: Fix Committed → In Progress
assignee: nobody → Geoff Teale (tealeg)
tags: removed: kanban-cross-team
James Page (james-page)
Changed in glance-simplestreams-sync (Juju Charms Collection):
status: In Progress → Fix Committed
importance: Undecided → High
James Page (james-page)
Changed in glance-simplestreams-sync (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in landscape:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.