[Update] Patching fails on neutron enviroments

Bug #1354050 reported by Tatyanka
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Critical
Sergey Vasilenko
5.0.x
Fix Committed
Critical
Sergey Vasilenko

Bug Description

Steps to reproduce:
1. Deploy simple with neutron on Ubuntu on 5.026
2. Upgrade to 5.1
3. Patch environment on 2014.1.1-5.0.2 release

Actual result:
Puppet fail on controller during patching with can not sync neutron db
http://paste.openstack.org/show/91583/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "410"
  build_id: "2014-08-07_16-17-02"
  astute_sha: "b52910642d6de941444901b0f20e95ebbcb2b2e9"
  fuellib_sha: "d699fc178559e98cfd7d53b58478b46553ffe39e"
  ostf_sha: "e33390c275e225d648b36997460dc29b1a3c20ae"
  nailgun_sha: "67c4f1c18ab0833175f6dc7f0f9c49c3eb722287"
  fuelmain_sha: "3d452c04345ca3b0ab0615661e8cf36e277c1d50"

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Revision history for this message
Dmitry Ilyin (idv1985) wrote :

2014-08-07T15:51:19.254323+01:00 notice: (/Stage[main]/Neutron::Server/Exec[upgrade_neutron_head]/returns) sqlalchemy.exc.OperationalError: (OperationalError) (1050, "Table 'networks' already exists") '\nCREATE TABLE networks (\n\ttenant_id VARCHAR(255), \n\tid VARCHAR(36) NOT NULL, \n\tname VARCHAR(255), \n\tstatus VARCHAR(16), \n\tadmin_state_up BOOL, \n\tshared BOOL, \n\tPRIMARY KEY (id), \n\tCHECK (admin_state_up IN (0, 1)), \n\tCHECK (shared IN (0, 1))\n)\n\n' ()

Changed in fuel:
status: New → Confirmed
Revision history for this message
Aleksey Kasatkin (alekseyk-ru) wrote :

One more.

Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

Together with other MOS team members we had a conversation in skype chat and identified both the problem and solution:

The reason of the failure is that Neutron's migrations were not run during 5.0 environment installation. I.e. the Neutron DB was created from metadata, and not by running 'neutron-db-manage upgrade head'. To patch environment to 5.0.2, fuel runs 'neutron-db-manage upgrade head' and it fails because the DB is already created, but it does not have a migrations stamp.

The solution is to run 'neutron-db-manage stamp head' prior to patching. Note that this command _must not_ be run before initial installation, only before the upgrade.

Changed in fuel:
status: Confirmed → Triaged
Revision history for this message
Ihor Kalnytskyi (ikalnytskyi) wrote :

Ok, `neutron-db-manage --config-file /etc/neutron/neutron.conf stamp head` fails with ImportError. It looks like we should set a full path to the plugin in `core_plugin` option before calling migration.

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Aleksandr Didenko (adidenko)
Revision history for this message
Aleksandr Didenko (adidenko) wrote :

Here's the exact error from `neutron-db-manage --config-file /etc/neutron/neutron.conf stamp head` command:

root@node-1:~# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini stamp head
No handlers could be found for logger "neutron.common.legacy"
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 169, in main
    CONF.command.func(config, CONF.command.name)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 88, in do_stamp
    sql=CONF.command.sql)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 60, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/alembic/command.py", line 257, in stamp
    script.run_env()
  File "/usr/lib/python2.7/dist-packages/alembic/script.py", line 193, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/dist-packages/alembic/util.py", line 177, in load_python_file
    module = load_module(module_id, path)
  File "/usr/lib/python2.7/dist-packages/alembic/compat.py", line 39, in load_module
    return imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/alembic_migrations/env.py", line 42, in <module>
    importutils.import_class(class_path)
  File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/importutils.py", line 33, in import_class
    traceback.format_exception(*sys.exc_info())))
ImportError: Class openvswitch cannot be found (['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/importutils.py", line 28, in import_class\n __import__(mod_str)\n', 'ValueError: Empty module name\n'])

Changed in fuel:
assignee: Aleksandr Didenko (adidenko) → Sergey Vasilenko (xenolog)
milestone: 5.1 → 5.0.1
Changed in fuel:
milestone: 5.0.1 → 5.0.2
Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/5.0.x
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/113290

Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 5.0.2 → 5.1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/113860

Changed in fuel:
status: Triaged → In Progress
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

this bug does not affect 5.1 as alembic stamp is already created and there are no upgrades between 5.0.x and 5.1.x .

Changed in fuel:
status: In Progress → Won't Fix
status: Won't Fix → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/113290
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=bb891a09dab29531ef6ace86be7f6bf81cc69a2d
Submitter: Jenkins
Branch: stable/5.0

commit bb891a09dab29531ef6ace86be7f6bf81cc69a2d
Author: Sergey Vasilenko <email address hidden>
Date: Mon Aug 11 19:25:39 2014 +0400

    Add notifying and ordering for neutron config options

    and database migration script

    Change-Id: I322b0b5d6090746ef6ec58a2b287eaae63e3f776
    Partial-Bug: #1354050

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Sergey Vasilenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/113860

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.