Juniper Openstack

API server errors preventing creation of routing/ service instance

Bug #1463642 reported by Vinod Nair
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
Critical
Hampapur Ajay
Juniper Openstack r2.20-fcs
Trunk
Fix Committed
Critical
Hampapur Ajay
Juniper Openstack r3.0-fcs

Bug Description

LBAAS netns and haproxy config is not getting created. Looks like the api server error is the one preventing the ceration of routing / service instances..

api server log is as below

http://cmbu-sv02.englab.juniper.net/pxe/Standard/vin/contrail-api.log

version : 2.02 build 46 Juno 14.04

root@cs-scale-1:/opt/contrail/utils# neutron lb-vip-list
+--------------------------------------+------+----------+----------+----------------+--------+
| id | name | address | protocol | admin_state_up | status |
+--------------------------------------+------+----------+----------+----------------+--------+
| 0afa003c-585f-4aa4-963a-47afd6eb626b | VIP1 | 11.1.0.3 | HTTP | True | ACTIVE |
+--------------------------------------+------+----------+----------+----------------+--------+

root@cs-scale-1:/opt/contrail/utils# neutron port-list | grep 11.1.0.3
| 93264673-316e-4f01-a308-8eb0688ce5ef | 0afa003c-585f-4aa4-963a-47afd6eb626b | 02:93:26:46:73:31 | {"subnet_id": "624809fa-8d39-4165-ac5d-2f0c8dd275eb", "ip_address": "11.1.0.3"} |

http://10.87.141.10:8081/analytics/uves/service-instances
[]

Api.log on one of the config nodes is as below

tailf /var/log/contrail/contrail-api.log
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_WARN]: VncApiError: Warning! InvalidRequestException(why='Keyspace names must be case-insensitively unique ("useragent" conflicts with "useragent")')
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_WARN]: VncApiError: Warning! InvalidRequestException(why='Cannot add already existing column family "useragent_keyval_table" to keyspace "useragent"')
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_WARN]: VncApiError: Warning! InvalidRequestException(why='Keyspace names must be case-insensitively unique ("config_db_uuid" conflicts with "config_db_uuid")')
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_WARN]: VncApiError: Warning! InvalidRequestException(why='Cannot add already existing column family "obj_uuid_table" to keyspace "config_db_uuid"')
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_WARN]: VncApiError: Warning! InvalidRequestException(why='Cannot add already existing column family "obj_fq_name_table" to keyspace "config_db_uuid"')
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_NOTICE]: VncApiError: Cassandra connection ESTABLISHED
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_NOTICE]: VncApiError: Connecting to zookeeper on 13.1.0.1:2181,13.1.0.2:2181,13.1.0.3:2181
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_NOTICE]: VncApiError: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@13.1.0.10:5673//']
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_ERR]: VncApiError: RabbitMQ connection down
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_NOTICE]: VncApiError: RabbitMQ connection ESTABLISHED <Connection: amqp://guest@13.1.0.10:5673// at 0x7fb90219a150>

See original description
Tags: api config
Vinod Nair (vinodnair)
tags: added: lbaas
no longer affects: juniperopenstack/r2.30
summary: - LBAAS: LBaas netns / haproxy.cfg is not getting created
+ LBaas netns / haproxy.cfg is not getting created
Vinod Nair (vinodnair)
summary: - LBaas netns / haproxy.cfg is not getting created
+ API server errors preventing creation of routing/ service instance
Vinod Nair (vinodnair)
description: updated
tags: added: api
removed: lbaas
Revision history for this message
Vinod Nair (vinodnair) wrote :

When following sequence of operations were done, the rabbimq drainer greenlet died forever. The rabbitmq drainer should keep running i.e. handle catch-all Exception

1. patch oslo for some backports on all controllers
2. restart rabbitmq server
3. restart all nova services
[ contrail-api was NOT restarted]

13.1.0.2 - - [2015-06-09 17:42:49] "GET /virtual-routers?detail=True HTTP/1.1" 401 192 0.022938
ERROR:cs-scale-2:contrail-api:Config:0:__default__ [SYS_ERR]: VncApiError: RabbitMQ connection down
WARNING:cs-scale-2:contrail-api:Config:0:__default__ [SYS_NOTICE]: VncApiError: RabbitMQ connection ESTABLISHED <Connection: amqp://guest@13.1.0.10:5673// at 0x7f085e208610>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line 112, in _connection_watch
    self._reconnect()
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line 99, in _reconnect
    callbacks=[self._subscribe])
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 357, in __init__
    self.revive(self.channel)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 369, in revive
    self.declare()
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 379, in declare
    queue.declare()
  File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 505, in declare
    self.queue_declare(nowait, passive=False)
  File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 531, in queue_declare
    nowait=nowait)
  File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 1258, in queue_declare
    (50, 11), # Channel.queue_declare_ok
  File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
    return self.dispatch_method(method_sig, args, content)
  File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 87, in dispatch_method
    return amqp_method(self, args)
  File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 241, in _close
    reply_code, reply_text, (class_id, method_id), ChannelError,
NotFound: Queue.declare: (404) NOT_FOUND - home node 'rabbit@cs-scale-1-ctrl' of durable queue 'vnc_config.cs-scale-2-9100' in vhost '/' is down or inaccessible
<Greenlet at 0x7f085e2017d0: <bound method VncServerKombuClient._connection_watch of <vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object at 0x7f085f3ac490>>> failed with NotFound

WARNING:keystoneclient.middleware.auth_token:Unable to find authentication token in headers
13.1.0.3 - - [2015-06-09 17:46:04] "GET / HTTP/1.1" 401 192 0.000700
13.1.0.3 - - [2015-06-09 17:46:04] "GET / HTTP/1.1" 200 11310 0.017901
13.1.0.3 - - [2015-06-09 17:46:04] "GET /physical-routers?count=False&detail=False HTTP/1.1" 200 132 0.002907
WARNING:keystoneclient.middleware.auth_token:Unable to find authentication token in headers
13.1.0.2 - - [2015-06-09 17:46:07] "GET / HTTP/1.1" 401 192 0.000702
13.1.0.2 - - [2015-06-09 17:46:08] "GET / HTTP/1.1" 200 11310 0.214082

Sachin Bansal (sbansal)
tags: added: config
OpenContrail Admin (ci-admin-f)
information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/11594
Submitter: Sachin Bansal (<email address hidden>)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.