neutron-api stays blocked: Services not running that should be: neutron-server

Bug #1970235 reported by Bas de Bruijne
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Neutron API Charm
New
Undecided
Unassigned

Bug Description

In testrun:
https://solutions.qa.canonical.com/testruns/testRun/ad989e74-d085-40bb-b79a-c39f51180e2a

with FCE console:
https://oil-jenkins.canonical.com/job/fce_build/1270//console

2 out of 3 neutron-api using stay blocked:
------------------------------------------------
neutron-api/0* active idle 0/lxd/6 10.246.166.106 9696/tcp Unit is ready
  filebeat/21 active idle 10.246.166.106 Filebeat ready.
  hacluster-neutron-api/0* active idle 10.246.166.106 Unit is ready and clustered
  landscape-client/21 maintenance idle 10.246.166.106 Need computer-title and juju-info to proceed
  logrotated/15 active idle 10.246.166.106 Unit is ready.
  neutron-api-mysql-router/0* active idle 10.246.166.106 Unit is ready
  nrpe/21 active idle 10.246.166.106 icmp,5666/tcp Ready (source version/commit cs-nrpe-...)
  public-policy-routing/3 active idle 10.246.166.106 Unit is ready
  telegraf/21 active idle 10.246.166.106 9103/tcp Monitoring neutron-api/0 (source version/commit 26e531a)
neutron-api/1 blocked idle 1/lxd/7 10.246.167.64 9696/tcp Services not running that should be: neutron-server
  filebeat/31 active idle 10.246.167.64 Filebeat ready.
  hacluster-neutron-api/1 active idle 10.246.167.64 Unit is ready and clustered
  landscape-client/35 maintenance idle 10.246.167.64 Need computer-title and juju-info to proceed
  logrotated/23 active idle 10.246.167.64 Unit is ready.
  neutron-api-mysql-router/1 active idle 10.246.167.64 Unit is ready
  nrpe/31 active idle 10.246.167.64 icmp,5666/tcp Ready (source version/commit cs-nrpe-...)
  public-policy-routing/6 active idle 10.246.167.64 Unit is ready
  telegraf/35 active idle 10.246.167.64 9103/tcp Monitoring neutron-api/1 (source version/commit 26e531a)
neutron-api/2 blocked idle 2/lxd/7 10.246.165.62 9696/tcp Services not running that should be: neutron-server
  filebeat/33 active idle 10.246.165.62 Filebeat ready.
  hacluster-neutron-api/2 active idle 10.246.165.62 Unit is ready and clustered
  landscape-client/33 maintenance idle 10.246.165.62 Need computer-title and juju-info to proceed
  logrotated/27 active idle 10.246.165.62 Unit is ready.
  neutron-api-mysql-router/2 active idle 10.246.165.62 Unit is ready
  nrpe/33 active idle 10.246.165.62 icmp,5666/tcp Ready (source version/commit cs-nrpe-...)
  public-policy-routing/9 active idle 10.246.165.62 Unit is ready
  telegraf/31 active idle 10.246.165.62 9103/tcp Monitoring neutron-api/2 (source version/commit 26e531a)
------------------------------------------------

The only thing I can see standing out in the logs is:
------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect
    return fn()
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 482, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 94, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 325, in __init__
    self.connect()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 599, in connect
    self._request_authentication()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 871, in _request_authentication
    auth_packet = self._process_auth(plugin_name, auth_packet)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 942, in _process_auth
    pkt = self._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1045, "Access denied for user 'neutron'@'10.246.170.69' (using password: YES)")
------------------------------------------------

This message is from about 3h before the end of the testrun, but it is possible that the units are not recovering from it.

Crashdumps can be found here:
https://oil-jenkins.canonical.com/artifacts/ad989e74-d085-40bb-b79a-c39f51180e2a/index.html

Tags: cdo-qa
tags: added: cdo-qa
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.