PCI device migration cannot continue with old deleted service records

Bug #1668310 reported by Dan Smith
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Dan Smith
Mitaka
Fix Released
High
Matt Riedemann
Newton
Fix Committed
High
Matt Riedemann
Ocata
Fix Committed
High
Matt Riedemann

Bug Description

If deleted service records are present in the database, the Service minimum version calculation should ignore them, but it does not. One manifestation of this is the PCI device migration from mitaka/newton will never complete, emitting an error message like this:

2017-02-27 07:40:19.665 ERROR nova.db.sqlalchemy.api [req-ad21480f-613a-445b-a913-c54532b64ffa None None] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version

Tags: db upgrades
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/438578

Changed in nova:
assignee: nobody → Dan Smith (danms)
status: New → In Progress
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → High
tags: added: db upgrades
Revision history for this message
Saverio Proto (zioproto) wrote :

I was able to workaround the problem like this:

delete from services where deleted != 0 ;
delete from aggregate_hosts where deleted !=0;
delete from aggregate_metadata where deleted !=0;
delete from aggregates where deleted !=0;

and then I also need nova to be at least at commit edbd3135f8087ea73dc73b06962bf8b869ee6f43

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/438628

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/438630

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/438632

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/438578
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c79770e615799cd4457ac603dcad4fb3452fe2bc
Submitter: Jenkins
Branch: master

commit c79770e615799cd4457ac603dcad4fb3452fe2bc
Author: Dan Smith <email address hidden>
Date: Mon Feb 27 07:52:29 2017 -0800

    Ignore deleted services in minimum version calculation

    When we go to detect the minimum version for a given service, we
    should ignore any deleted services. Without this, we will return
    the minimum version of all records, including those that have been
    deleted with "nova service-delete". This patch filters deleted
    services from the query.

    Closes-Bug: #1668310
    Change-Id: Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/438630
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1ad5c7305c37079ced24bf623810e63d5eac2661
Submitter: Jenkins
Branch: stable/newton

commit 1ad5c7305c37079ced24bf623810e63d5eac2661
Author: Dan Smith <email address hidden>
Date: Mon Feb 27 07:52:29 2017 -0800

    Ignore deleted services in minimum version calculation

    When we go to detect the minimum version for a given service, we
    should ignore any deleted services. Without this, we will return
    the minimum version of all records, including those that have been
    deleted with "nova service-delete". This patch filters deleted
    services from the query.

    Closes-Bug: #1668310
    Change-Id: Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b
    (cherry picked from commit c79770e615799cd4457ac603dcad4fb3452fe2bc)
    (cherry picked from commit a1dd547d3ba7d9cf5b73da1ce9668e412501ace1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/mitaka)

Reviewed: https://review.openstack.org/438632
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2e05c82b807624b8420df4c94d68b75563b230d2
Submitter: Jenkins
Branch: stable/mitaka

commit 2e05c82b807624b8420df4c94d68b75563b230d2
Author: Dan Smith <email address hidden>
Date: Mon Feb 27 07:52:29 2017 -0800

    Ignore deleted services in minimum version calculation

    When we go to detect the minimum version for a given service, we
    should ignore any deleted services. Without this, we will return
    the minimum version of all records, including those that have been
    deleted with "nova service-delete". This patch filters deleted
    services from the query.

    Conflicts:
            nova/db/sqlalchemy/api.py
            nova/tests/unit/db/test_db_api.py

    NOTE(mriedem): The conflicts are due to not having change
    11cb56a2243faa9f2614a8f9a9a84603bc91d6b1 in Mitaka.

    Closes-Bug: #1668310
    Change-Id: Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b
    (cherry picked from commit c79770e615799cd4457ac603dcad4fb3452fe2bc)
    (cherry picked from commit a1dd547d3ba7d9cf5b73da1ce9668e412501ace1)
    (cherry picked from commit 1ad5c7305c37079ced24bf623810e63d5eac2661)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/438628
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a1dd547d3ba7d9cf5b73da1ce9668e412501ace1
Submitter: Jenkins
Branch: stable/ocata

commit a1dd547d3ba7d9cf5b73da1ce9668e412501ace1
Author: Dan Smith <email address hidden>
Date: Mon Feb 27 07:52:29 2017 -0800

    Ignore deleted services in minimum version calculation

    When we go to detect the minimum version for a given service, we
    should ignore any deleted services. Without this, we will return
    the minimum version of all records, including those that have been
    deleted with "nova service-delete". This patch filters deleted
    services from the query.

    Closes-Bug: #1668310
    Change-Id: Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b
    (cherry picked from commit c79770e615799cd4457ac603dcad4fb3452fe2bc)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 13.1.3

This issue was fixed in the openstack/nova 13.1.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.1

This issue was fixed in the openstack/nova 15.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.5

This issue was fixed in the openstack/nova 14.0.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.0.0b1

This issue was fixed in the openstack/nova 16.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.