VNC stops working in rolling upgrade by default

Bug #1798188 reported by Mohammed Naser
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
melanie witt
Rocky
Fix Committed
High
melanie witt

Bug Description

During a rolling upgrade, once the control plane is upgraded and running on Rocky (but computes still in Queens), the consoles will stop working.

It is not obvious however it seems that the following is missing:

```
[workarounds]
enable_consoleauth = True
```

There isn't a really obvious document or anything explaining this, leaving the user confused

melanie witt (melwitt)
Changed in nova:
assignee: nobody → melanie witt (melwitt)
importance: Undecided → High
status: New → Confirmed
tags: added: console upgrade
Revision history for this message
Matt Riedemann (mriedem) wrote :

One suggestion in IRC today was that we could add a "nova-status upgrade check" which iterates the cell DBs looking to see if there are any non-deleted/disabled nova-consoleauth services table records and if so, check to see if there are no console_auth_tokens entries in that DB and if [workarounds]/enable_consoleauth is False, fail (or warn?) the status check b/c it means that cell is using nova-consoleauth but doesn't have tokens in the DB so it needs to run the consoleauth service.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/611214

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/611218

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/613174

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by melanie witt (<email address hidden>) on branch: master
Review: https://review.openstack.org/611218
Reason: This should go direct to the stable branch and that is proposed here:

https://review.openstack.org/613174

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/614302

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/611214
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d2535b0261205ed80e1731451496caff19ef3f63
Submitter: Zuul
Branch: master

commit d2535b0261205ed80e1731451496caff19ef3f63
Author: melanie witt <email address hidden>
Date: Tue Oct 16 20:29:18 2018 +0000

    Add nova-status upgrade check for consoles

    This will check if a deployment is currently using consoles and warns
    the operator to set [workarounds]enable_consoleauth = True on their
    console proxy host if they are performing a rolling upgrade which is
    not yet complete.

    Partial-Bug: #1798188

    Change-Id: Idd6079ce4038d6f19966e98bcc61422b61b3636b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/613174
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d362e4285137de63e14f0fe8e24fa874a975660b
Submitter: Zuul
Branch: stable/rocky

commit d362e4285137de63e14f0fe8e24fa874a975660b
Author: melanie witt <email address hidden>
Date: Wed Oct 17 02:54:38 2018 +0000

    Add rolling upgrade info to enable_consoleauth workaround reno

    This explains how the [workarounds]enable_consoleauth option needs to
    used if an operator is performing a live, rolling upgrade.

    Closes-Bug: #1798188

    Change-Id: Ie637b4871df8b870193b5bc07eece15c03860c06

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/614302
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2147093a966864434ab90d9167324c561e260fb5
Submitter: Zuul
Branch: stable/rocky

commit 2147093a966864434ab90d9167324c561e260fb5
Author: melanie witt <email address hidden>
Date: Tue Oct 16 20:29:18 2018 +0000

    Add nova-status upgrade check for consoles

    This will check if a deployment is currently using consoles and warns
    the operator to set [workarounds]enable_consoleauth = True on their
    console proxy host if they are performing a rolling upgrade which is
    not yet complete.

    Partial-Bug: #1798188

     Conflicts:
     doc/source/cli/nova-status.rst

    NOTE(melwitt): The conflict is from the absence of the
    **19.0.0 (Stein)** section in Rocky and the absence of the placement
    API check to require version 1.30 in Rocky.

    Change-Id: Idd6079ce4038d6f19966e98bcc61422b61b3636b
    (cherry picked from commit d2535b0261205ed80e1731451496caff19ef3f63)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.3

This issue was fixed in the openstack/nova 18.0.3 release.

Revision history for this message
melanie witt (melwitt) wrote :

This is considered fixed since https://review.openstack.org/611214 merged, since an update to the rocky release note went directly to the stable/rocky branch and not to master. The patch used "Partial-Bug" in the commit message because it was intended to go alongside a release note patch to close out the bug, but the release note patch was stable-only and went only to the stable/rocky branch.

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/605154
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=983e6ea5518dee956d8054cd67dddbe98eab9f78
Submitter: Zuul
Branch: master

commit 983e6ea5518dee956d8054cd67dddbe98eab9f78
Author: melanie witt <email address hidden>
Date: Tue Sep 25 17:12:16 2018 +0000

    Restore nova-consoleauth to install docs

    The installation of the nova-consoleauth service was erroneously
    removed from the docs prematurely. The nova-consoleauth service
    is still being used in Rocky, with the removal being possible in
    Stein.

    This should have been fixed as part of change
    Ibbdc7c50c312da2acc59dfe64de95a519f87f123 but was missed.

    This is also related to the release note update in Rocky
    under change Ie637b4871df8b870193b5bc07eece15c03860c06.

    Co-Authored-By: Matt Riedemann <email address hidden>

    Closes-Bug: #1793255
    Related-Bug: #1798188

    Change-Id: Ied268da9e70bd2807c2dfe7a479181fbec52979d

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/630997

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/630997
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7a0203c9ad1b10596f5de0d0954ee3ccb356d685
Submitter: Zuul
Branch: stable/rocky

commit 7a0203c9ad1b10596f5de0d0954ee3ccb356d685
Author: melanie witt <email address hidden>
Date: Tue Sep 25 17:12:16 2018 +0000

    Restore nova-consoleauth to install docs

    The installation of the nova-consoleauth service was erroneously
    removed from the docs prematurely. The nova-consoleauth service
    is still being used in Rocky, with the removal being possible in
    Stein.

    This should have been fixed as part of change
    Ibbdc7c50c312da2acc59dfe64de95a519f87f123 but was missed.

    This is also related to the release note update in Rocky
    under change Ie637b4871df8b870193b5bc07eece15c03860c06.

    Co-Authored-By: Matt Riedemann <email address hidden>

    Closes-Bug: #1793255
    Related-Bug: #1798188

    Change-Id: Ied268da9e70bd2807c2dfe7a479181fbec52979d
    (cherry picked from commit 983e6ea5518dee956d8054cd67dddbe98eab9f78)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.