Juju Charms Collection
neutron-api package

Neutron db migration fails, lbaas_members doesn't exist

Bug #1596594 reported by Peter Sabaini
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Neutron API Charm
Expired
High
Unassigned
neutron-api (Juju Charms Collection)
Invalid
High
James Page
Juju Charms Collection 17.01

Bug Description

Upgrading to Mitaka fails in neutron db migration. Upgrade performed via setting openstack-origin.

Running db migration by hand errors with:

# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head
No handlers could be found for logger "oslo_config.cfg"
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
  Running upgrade for neutron ...
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
  OK
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
  Running upgrade for neutron-fwaas ...
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
  OK
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
  Running upgrade for neutron-lbaas ...
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 3345facd0452 -> 4a408dd491c2, Addition of Name column to lbaas_members and lbaas_healthmonitors table
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 749, in main
    return_val |= bool(CONF.command.func(config, CONF.command.name))
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 225, in do_upgrade
    desc=branch, sql=CONF.command.sql)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 127, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/alembic/command.py", line 174, in upgrade
    script.run_env()
  File "/usr/lib/python2.7/dist-packages/alembic/script/base.py", line 397, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/dist-packages/alembic/util/pyfiles.py", line 81, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/dist-packages/alembic/util/compat.py", line 79, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/dist-packages/neutron_lbaas/db/migration/alembic_migrations/env.py", line 85, in <module>
    run_migrations_online()
  File "/usr/lib/python2.7/dist-packages/neutron_lbaas/db/migration/alembic_migrations/env.py", line 76, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/lib/python2.7/dist-packages/alembic/runtime/environment.py", line 797, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/lib/python2.7/dist-packages/alembic/runtime/migration.py", line 312, in run_migrations
    step.migration_fn(**kw)
  File "/usr/lib/python2.7/dist-packages/neutron_lbaas/db/migration/alembic_migrations/versions/mitaka/expand/4a408dd491c2_UpdateName.py", line 35, in upgrade
    op.add_column(table, sa.Column('name', sa.String(255), nullable=True))
  File "<string>", line 8, in add_column
  File "<string>", line 3, in add_column
  File "/usr/lib/python2.7/dist-packages/alembic/operations/ops.py", line 1535, in add_column
    return operations.invoke(op)
  File "/usr/lib/python2.7/dist-packages/alembic/operations/base.py", line 318, in invoke
    return fn(self, operation)
  File "/usr/lib/python2.7/dist-packages/alembic/operations/toimpl.py", line 123, in add_column
    schema=schema
  File "/usr/lib/python2.7/dist-packages/alembic/ddl/impl.py", line 172, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/usr/lib/python2.7/dist-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 968, in _execute_ddl
    compiled
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 158, in execute
    result = self._query(query)
  File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 308, in _query
    conn.query(q)
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 820, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1002, in _read_query_result
    result.read()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1285, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 966, in _read_packet
    packet.check_error()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
    raise errorclass(errno, errorvalue)
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'neutron.lbaas_members' doesn't exist") [SQL: u'ALTER TABLE lbaas_members ADD COLUMN name VARCHAR(255)']

Revision history for this message
JuanJo Ciarlante (jjo) wrote :

Please note this is quite critical, as it breaks L->M db migrations.
A very dirty hack to skip lbaas migrations entirely:
http://paste.ubuntu.com/20364983/
, will let migrations succeed with:

neutron-db-manage upgrade heads

, above will obviously spoil lbaas mitaka service.

James Page (james-page)
Changed in neutron-api (Juju Charms Collection):
assignee: nobody → James Page (james-page)
importance: Undecided → High
milestone: none → 16.07
Revision history for this message
James Page (james-page) wrote :

OK - so I just tried to reproduce this problem, but my upgrade went OK; this is what I did:

a) deployed a fresh trusty-liberty cloud
b) ran tempest smoke against it to generate some database state (smoke passed in full)
c) upgraded keystone using openstack-origin=cloud:trusty-mitaka
d) upgraded neutron-api using openstack-origin=cloud:trusty-mitaka

The DB migration in neutron-api succeeded OK - so I'm a little puzzled as to why I don't see the same problem. Did the impacted deployments start on an earlier version of OpenStack? I only tested a single step, and this might be a multi-step problem.

Changed in neutron-api (Juju Charms Collection):
status: New → Incomplete
Revision history for this message
James Page (james-page) wrote :
Download full text (9.3 KiB)

Log from upgrade:

unit-neutron-api-0[3044]: 2016-07-22 07:59:44 INFO unit.neutron-api/0.juju-log server.go:268 Migrating the neutron database.
unit-neutron-api-0[3044]: 2016-07-22 07:59:46 INFO unit.neutron-api/0.config-changed logger.go:40 No handlers could be found for logger "oslo_config.cfg"
unit-neutron-api-0[3044]: 2016-07-22 07:59:48 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Context impl MySQLImpl.
unit-neutron-api-0[3044]: 2016-07-22 07:59:48 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Will assume non-transactional DDL.
unit-neutron-api-0[3044]: 2016-07-22 07:59:48 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Context impl MySQLImpl.
unit-neutron-api-0[3044]: 2016-07-22 07:59:48 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Will assume non-transactional DDL.
unit-neutron-api-0[3044]: 2016-07-22 07:59:48 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Running upgrade 34af2b5c5a59 -> 59cb5b6cf4d, Add availability zone
unit-neutron-api-0[3044]: 2016-07-22 07:59:48 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Running upgrade 59cb5b6cf4d -> 13cfb89f881a, add is_default to subnetpool
unit-neutron-api-0[3044]: 2016-07-22 07:59:48 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Running upgrade 13cfb89f881a -> 32e5974ada25, Add standard attribute table
unit-neutron-api-0[3044]: 2016-07-22 07:59:48 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Running upgrade 32e5974ada25 -> ec7fcfbf72ee, Add network availability zone
unit-neutron-api-0[3044]: 2016-07-22 07:59:48 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Running upgrade ec7fcfbf72ee -> dce3ec7a25c9, Add router availability zone
unit-neutron-api-0[3044]: 2016-07-22 07:59:48 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Running upgrade dce3ec7a25c9 -> c3a73f615e4, Add ip_version to AddressScope
unit-neutron-api-0[3044]: 2016-07-22 07:59:49 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Running upgrade c3a73f615e4 -> 659bf3d90664, Add tables and attributes to support external DNS integration
unit-neutron-api-0[3044]: 2016-07-22 07:59:49 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Running upgrade 659bf3d90664 -> 1df244e556f5, add_unique_ha_router_agent_port_bindings
unit-neutron-api-0[3044]: 2016-07-22 07:59:49 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Running upgrade 1df244e556f5 -> 19f26505c74f, Auto Allocated Topology - aka Get-Me-A-Network
unit-neutron-api-0[3044]: 2016-07-22 07:59:49 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migration] Running upgrade 19f26505c74f -> 15be73214821, add dynamic routing model data
unit-neutron-api-0[3044]: 2016-07-22 07:59:49 INFO unit.neutron-api/0.config-changed logger.go:40 INFO [alembic.runtime.migratio...

Read more...

Liam Young (gnuoy)
Changed in neutron-api (Juju Charms Collection):
milestone: 16.07 → 16.10
James Page (james-page)
Changed in neutron-api (Juju Charms Collection):
milestone: 16.10 → 17.01
Revision history for this message
Darryl Weaver (dweaver) wrote :

Just to confirm, I just tripped over this bug too, but this was a multiple upgraded Openstack.

We started on Juju 1.25 and deployed Kilo using old charms to match our old staging deployment.
We then upgraded the charms and upgraded to Liberty, went OK, with some minor issues.
We then updated to the latest charms and started the upgrade to Mitaka.
We then tripped over this one after upgrading Keystone, cinder and glance.
We stopped here to review and found this bug which is exactly what we found, i.e. the migration process fails as there is no table lbaas_members in the neutron database.

James Page (james-page)
Changed in charm-neutron-api:
assignee: nobody → James Page (james-page)
importance: Undecided → High
status: New → Incomplete
Changed in neutron-api (Juju Charms Collection):
status: Incomplete → Invalid
Revision history for this message
Vijay M (vj.m) wrote :

Didn't see any resolution or bug fixes mentioned after comment #4, as I encountered the same issue when upgrading from Kilo -> Liberty -> Mitaka using Juju 1.25.*.

# juju ssh neutron-api/0 sudo neutron-db-manage upgrade heads
...
...
Running upgrade 3345facd0452 -> 4a408dd491c2, Addition of Name column to lbaas_members and lbaas_healthmonitors table
...
...
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'neutron.lbaas_members' doesn't exist") [SQL: u'ALTER TABLE lbaas_members ADD COLUMN name VARCHAR(255)']

Revision history for this message
Vijay M (vj.m) wrote :

Looks like Lbaas V2 creates these tables

neutron-lbaas/neutron_lbaas/db/migration/alembic_migrations/versions/lbaasv2.py

op.create_table(
        u'lbaas_members',
....
op.create_table(
        u'lbaas_healthmonitors',
....

I don't have lbaas V2 enabled in the environment, will try Lbaas V2 and see if that resolves the issue.

Revision history for this message
Peter Sabaini (peter-sabaini) wrote :

Very very late reply: yes, my original report was for a multi-upgrade cloud as well, starting from Kilo

Revision history for this message
Drew Freiberger (afreiberger) wrote :

I just ran into this bug On a cloud that was liberty, now mitaka. Was 17.02 charms, I stepped the openstack environment up to 17.08 charms, then I switched to the 17.11 keystone and mysql charms, and neutron-api leader went crazy with this while at 17.08 charms connecting to percona-cluster 17.11 (current cs:percona-cluster charm) in the shared-db-relation-changed hook.

I'm not sure why this is happening now, because the openstack-origin has stayed cloud:trusty-mitaka through the 17.02 to 17.08 charm upgrade and now with the 17.11 charms, still not doing an upgrade.

Maybe there's something that's an artifact of the prior upgrade that triggers the need to upgrade the db for neutron in the charm.

Drew Freiberger (afreiberger)
Changed in charm-neutron-api:
status: Incomplete → Confirmed
Alex Kavanagh (ajkavanagh)
tags: added: openstack-upgrade
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Okay, so this is a very tricky one. I'm setting it back to incomplete as there still doesn't appear to be a reproducible way of triggering the issue. i.e. it may be packaging, or a sequence of packaging, or a sequence of charm upgrades, or sequence of openstack upgrades, or something else?

Drew, et al, what did you do to mitigate the situation; i.e. how did you recover it? I'm wondering whether we can put a check in and bail before it is attempted and/or putting a known issue in the release notes on how to handle the situation. Thanks very much!

Changed in charm-neutron-api:
assignee: James Page (james-page) → nobody
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack neutron-api charm because there has been no activity for 60 days.]

Changed in charm-neutron-api:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.