Default value of the Cluster.replaced_deployment_info field was changed without appropriate migration

Bug #1568100 reported by Ilya Kharin
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Georgy Kibardin
Mitaka
Fix Released
Critical
Georgy Kibardin

Bug Description

The default value of the Cluster.replaced_deployment_info field was changed from '{}' to '[]' without an appropriate DB migration script during 9.0 release [1] and [2]. Also this field was defined as MutableList and this type does to accept dict values. The problem appears after migration of the database in case when there is at least one cluster created in previous releases < 9.0 that does not have replaced deployment information.

Steps to reproduce:
1. Create environment using the 8.0 release
2. Upgrade master node to the 9.0 release (involve applying of migration scripts to synchronize the database to 9.0)
3. Try to get a list of environments using the fuel client:
    fuel environment

Expected results:
A list of existing environments will be printed.

Actual result:
This command fails with the error:

    500 Server Error: Internal Server Error for url: http://10.20.0.2:8000/api/v1/clusters/ (Attribute 'replaced_deployment_info' does not accept objects of type <type 'dict'>)

The traceback from /var/log/nailgun/app.log:

    2016-04-08 19:01:49.225 ERROR [7f2b8f602880] (base) Unexpected exception occured
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line 285, in content_json
        resp = func(cls, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line 446, in GET
        return self.collection.to_json(q)
      File "/usr/lib/python2.7/site-packages/nailgun/objects/base.py", line 437, in to_json
        iterable=iterable
      File "/usr/lib/python2.7/site-packages/nailgun/objects/base.py", line 421, in to_list
        use_iterable
      File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 86, in instances
        util.raise_from_cause(err)
      File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb, cause=cause)
      File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 71, in instances
        rows = [proc(row) for row in fetch]
      File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 432, in _instance
        state.manager.dispatch.load(state, context)
      File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
        fn(*args, **kw)
      File "/usr/lib64/python2.7/site-packages/sqlalchemy/ext/mutable.py", line 451, in load
        val = cls.coerce(key, val)
      File "/usr/lib/python2.7/site-packages/nailgun/db/sqlalchemy/models/mutable.py", line 141, in coerce
        return MutableBase.coerce(key, value)
      File "/usr/lib64/python2.7/site-packages/sqlalchemy/ext/mutable.py", line 403, in coerce
        raise ValueError(msg % (key, type(value)))
    ValueError: Attribute 'replaced_deployment_info' does not accept objects of type <type 'dict'>

Reproducibility:
This bug reproduces constantly if there is at least one environment that was created in the previous releases (<9.0) without the replaced deployment information.

Workaround:
Database entries can be fixed by the next SQL command:
    UPDATE clusters SET replaced_deployment_info = '[]' WHERE replaced_deployment_info = '{}';

Impact:
This bug is a blocker for the upgrade procedure of the master node from 8.0 to 9.0.

Links:
[1] The default value was changed to '[]' and the type was changed to MutableList in the next change request: https://review.openstack.org/#/c/249723/
[2] This change was made as a part of this bug: https://bugs.launchpad.net/fuel/+bug/1482658

Ilya Kharin (akscram)
description: updated
description: updated
Changed in fuel:
importance: Undecided → Critical
status: New → Confirmed
milestone: none → 9.0
assignee: nobody → Fuel Python Team (fuel-python)
tags: added: area-python feature-upgrader
tags: added: feature-upgrade
removed: feature-upgrader
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Georgy Kibardin (gkibardin)
milestone: 9.0 → 10.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/305004

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310730

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/mitaka)

Change abandoned by Georgy Kibardin (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/310730

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/305004
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=bb1149961a07943b7343ac53ef60d301a36988d1
Submitter: Jenkins
Branch: master

commit bb1149961a07943b7343ac53ef60d301a36988d1
Author: Georgy Kibardin <email address hidden>
Date: Wed Apr 13 10:15:46 2016 +0300

    Upgrade existing values for Cluster.replaced_deployment_info

    Expected value type for subj. has changed from dict to list
    in 9.0. From now migration takes into account not only 5.0
    databases.
    Original review is here https://review.openstack.org/249723

    Change-Id: Id1954d9607bc253c325e335e0dab0965d245c665
    Closes-Bug: #1568100

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/mitaka)

Change abandoned by Georgy Kibardin (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/310730

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/310730
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=d896f0824e87f77ea9e5d81ba916bedf3513f12c
Submitter: Jenkins
Branch: stable/mitaka

commit d896f0824e87f77ea9e5d81ba916bedf3513f12c
Author: Georgy Kibardin <email address hidden>
Date: Wed Apr 13 10:15:46 2016 +0300

    Upgrade existing values for Cluster.replaced_deployment_info

    Expected value type for subj. has changed from dict to list
    in 9.0. From now migration takes into account not only 5.0
    databases.
    Original review is here https://review.openstack.org/249723

    Change-Id: Id1954d9607bc253c325e335e0dab0965d245c665
    Closes-Bug: #1568100
    (cherry picked from commit bb1149961a07943b7343ac53ef60d301a36988d1)

tags: added: on-verification
tags: removed: on-verification
tags: added: dev-to-verify
Revision history for this message
ZHI BING WANG (zwang) wrote :

We have the same issue at a major customer

Revision history for this message
Georgy Kibardin (gkibardin) wrote :

ZHI BING WANG, have you tried the workaround proposed above?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.