AIO-SX reboots after change OAM ip address

Bug #1927275 reported by Andre Kantek
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
High
Andre Kantek

Bug Description

Brief Description
-----------------
AIO-SX reboots after change OAM ip address
Change OAM ip address of the AIO-SX system results in a reboot

Severity
--------
Major

Steps to Reproduce
------------------
Modify oam_ip of AIO-SX controller
ex:
system oam-modify oam_ip=2620:10a:a001:a103::1148

Expected Behavior
-----------------
Changing oam_ip of AIO-SX system should not result in a reboot

Actual Behavior
---------------
Changing oam_ip of AIO-SX system results in a reboot

Reproducibility
---------------
100% (3 of 3)

System Configuration
--------------------
AIO-SX
WCP112 (IPv6)

Last Pass
---------
2021-04-30

Timestamp/Logs
--------------
2021-05-05T13:02:59.000

Test Activity
-------------
Regression Testing

Workaround
----------
not known
Although the controller does retain the "new" oam_ip after it reboots

CVE References

Andre Kantek (akantek)
Changed in starlingx:
assignee: nobody → Andre Kantek (akantek)
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Screening: stx.5.0 / high - issue introduced by a recent change for https://bugs.launchpad.net/starlingx/+bug/1926288 which was also cherrypicked to the r/stx.5.0 release, so the fix will need to be cherrypicked as well.

Changed in starlingx:
importance: Undecided → High
tags: added: stx.5.0 stx.networking
tags: added: stx.6.0
Ghada Khalil (gkhalil)
Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/789946

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/789947

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-puppet (master)

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/789946
Committed: https://opendev.org/starlingx/stx-puppet/commit/c120fb798091db9fb756e51b895dccfa8d80a947
Submitter: "Zuul (22348)"
Branch: master

commit c120fb798091db9fb756e51b895dccfa8d80a947
Author: Andre Fernando Zanella Kantek <email address hidden>
Date: Wed May 5 17:30:19 2021 -0400

    AIO-SX reboots after change OAM ip address

    On HW tests, it was detected that openstack-endpoints restart was
    happening at the same as the service-manager restart, this creating
    a conflict that preventing SM services to reach enabled-active.
    This was provoking the reboot.

    The correction creates openstack::keystone::endpoint::runtime::post
    class to be executed the post stage and not on the main stage, to
    avoid conflict with service-manager

    Also marking platform::network::runtime to be run at the pre stage
    to avoid some encountered apply errors related to the delay of
    haproxy bringup due to the lack of the IP address on the interface
    as it was only configured later. This way the other restarted
    services will have the address on the interface as restart happens

    Tested on AIO-SX, by monitoring manifest apply and validating that
    no reboot happens

    Closes-Bug: 1927275

    Signed-off-by: Andre Fernando Zanella Kantek <email address hidden>
    Change-Id: Ia70a3395753e43b3c1e2c037818c8c23e4ec0fd6

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/c/starlingx/config/+/789947
Committed: https://opendev.org/starlingx/config/commit/3e9982bab088df428e786d5fdf2515dc21f189fa
Submitter: "Zuul (22348)"
Branch: master

commit 3e9982bab088df428e786d5fdf2515dc21f189fa
Author: Andre Fernando Zanella Kantek <email address hidden>
Date: Wed May 5 17:21:35 2021 -0400

    AIO-SX reboots after change OAM ip address

    On HW tests, it was detected that openstack-endpoints restart was
    happening at the same as the service-manager restart, this creating
    a conflict that preventing SM services to reach enabled-active.
    This was provoking the reboot.

    The correction add a class to execute openstack-endpoint runtime
    restart on the post stage on puppet, avoiding to be run as SM is
    restarted.

    Tested on AIO-SX, by monitoring manifest apply and validating that
    no reboot happens

    Closes-Bug: 1927275
    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/789946

    Signed-off-by: Andre Fernando Zanella Kantek <email address hidden>
    Change-Id: I9f547fbcc73ba5fea077c764a4a9282a02ac71c6

Ghada Khalil (gkhalil)
tags: added: stx.cherrypickneeded
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (r/stx.5.0)

Fix proposed to branch: r/stx.5.0
Review: https://review.opendev.org/c/starlingx/config/+/790259

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (r/stx.5.0)

Fix proposed to branch: r/stx.5.0
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/790260

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-puppet (r/stx.5.0)

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/790260
Committed: https://opendev.org/starlingx/stx-puppet/commit/516769217c4d913f8dee53714fef9902a5504696
Submitter: "Zuul (22348)"
Branch: r/stx.5.0

commit 516769217c4d913f8dee53714fef9902a5504696
Author: Andre Fernando Zanella Kantek <email address hidden>
Date: Wed May 5 17:30:19 2021 -0400

    AIO-SX reboots after change OAM ip address

    On HW tests, it was detected that openstack-endpoints restart was
    happening at the same as the service-manager restart, this creating
    a conflict that preventing SM services to reach enabled-active.
    This was provoking the reboot.

    The correction creates openstack::keystone::endpoint::runtime::post
    class to be executed the post stage and not on the main stage, to
    avoid conflict with service-manager

    Also marking platform::network::runtime to be run at the pre stage
    to avoid some encountered apply errors related to the delay of
    haproxy bringup due to the lack of the IP address on the interface
    as it was only configured later. This way the other restarted
    services will have the address on the interface as restart happens

    Tested on AIO-SX, by monitoring manifest apply and validating that
    no reboot happens

    Closes-Bug: 1927275

    Signed-off-by: Andre Fernando Zanella Kantek <email address hidden>
    Change-Id: Ia70a3395753e43b3c1e2c037818c8c23e4ec0fd6
    (cherry picked from commit c120fb798091db9fb756e51b895dccfa8d80a947)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (r/stx.5.0)

Reviewed: https://review.opendev.org/c/starlingx/config/+/790259
Committed: https://opendev.org/starlingx/config/commit/ac3fb24b1d372b9797642862962b9403539980d1
Submitter: "Zuul (22348)"
Branch: r/stx.5.0

commit ac3fb24b1d372b9797642862962b9403539980d1
Author: Andre Fernando Zanella Kantek <email address hidden>
Date: Wed May 5 17:21:35 2021 -0400

    AIO-SX reboots after change OAM ip address

    On HW tests, it was detected that openstack-endpoints restart was
    happening at the same as the service-manager restart, this creating
    a conflict that preventing SM services to reach enabled-active.
    This was provoking the reboot.

    The correction add a class to execute openstack-endpoint runtime
    restart on the post stage on puppet, avoiding to be run as SM is
    restarted.

    Tested on AIO-SX, by monitoring manifest apply and validating that
    no reboot happens

    Closes-Bug: 1927275
    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/789946

    Signed-off-by: Andre Fernando Zanella Kantek <email address hidden>
    Change-Id: I9f547fbcc73ba5fea077c764a4a9282a02ac71c6
    (cherry picked from commit 3e9982bab088df428e786d5fdf2515dc21f189fa)

Bill Zvonar (billzvonar)
tags: added: in-r-stx50
removed: stx.cherrypickneeded
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792009

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on stx-puppet (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792009

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792013

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on stx-puppet (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792013

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792018

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on stx-puppet (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792018

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792029

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/config/+/793460

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/config/+/793696

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/config/+/794611

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-puppet (f/centos8)
Download full text (48.0 KiB)

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/792029
Committed: https://opendev.org/starlingx/stx-puppet/commit/2b026190a3cb6d561b6ec4a46dfb3add67f1fa69
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 3e3940824dfb830ebd39fd93265b983c6a22fc51
Author: Dan Voiculeasa <email address hidden>
Date: Thu May 13 18:03:45 2021 +0300

    Enable kubelet support for pod pid limit

    Enable limiting the number of pids inside of pods.

    Add a default value to protect against a missing value.
    Default to 750 pids limit to align with service parameter default
    value for most resource consuming StarlingX optional app (openstack).
    In fact any value above service parameter minimum value is good for the
    default.

    Closes-Bug: 1928353
    Signed-off-by: Dan Voiculeasa <email address hidden>
    Change-Id: I10c1684fe3145e0a46b011f8e87f7a23557ddd4a

commit 0c16d288fbc483103b7ba5dad7782e97f59f4e17
Author: Jessica Castelino <email address hidden>
Date: Tue May 11 10:21:57 2021 -0400

    Safe restart of the etcd SM service in etcd upgrade runtime class

    While upgrading the central cloud of a DC system, activation failed
    because there was an unexpected SWACT to controller-1. This was due
    to the etcd upgrade script. Part of this script runs the etcd
    manifest. This triggers a reload/restart of the etcd service. As this
    is done outside of the sm, sm saw the process failure and triggered
    the SWACT.

    This commit modifies platform::etcd::upgrade::runtime puppet class
    to do a safe restart of the etcd SM service and thus, solve the
    issue.

    Change-Id: I3381b6976114c77ee96028d7d96a00302ad865ec
    Signed-off-by: Jessica Castelino <email address hidden>
    Closes-Bug: 1928135

commit eec3008f600aeeb69a42338ed44332228a862d11
Author: Mihnea Saracin <email address hidden>
Date: Mon May 10 13:09:52 2021 +0300

    Serialize updates to global_filter in the AIO manifest

    Right now, looking at the aio manifest:
    https://review.opendev.org/c/starlingx/stx-puppet/+/780600/15/puppet-manifests/src/manifests/aio.pp
    there are 3 classes that update
    in parallel the lvm global_filter:
    - include ::platform::lvm::controller
    - include ::platform::worker::storage
    - include ::platform::lvm::compute
    And this generates some errors.

    We fix this by adding dependencies between the above classes
    in order to update the global_filter in a serial mode.

    Closes-Bug: 1927762
    Signed-off-by: Mihnea Saracin <email address hidden>
    Change-Id: If6971e520454cdef41138b2f29998c036d8307ff

commit 97371409b9b2ae3f0db6a6a0acaeabd74927160e
Author: Steven Webster <email address hidden>
Date: Fri May 7 15:33:43 2021 -0400

    Add SR-IOV rate-limit dependency

    Currently, the binding of an SR-IOV virtual function (VF) to a
    driver has a dependency on platform::networking. This is needed
    to ensure that SR-IOV is enabled (VFs created) before actually
    doing the bind.

    This dependency does not exist for configuring the VF rate-limits
    however. There is a cha...

tags: added: in-f-centos8
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/config/+/794906

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on config (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/config/+/794611

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (f/centos8)
Download full text (147.3 KiB)

Reviewed: https://review.opendev.org/c/starlingx/config/+/794906
Committed: https://opendev.org/starlingx/config/commit/75758b37a5a23c8811355b67e2a430a1713cd85b
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 9e420d9513e5fafb1df4d29567bc299a9e04d58d
Author: Bin Qian <email address hidden>
Date: Mon May 31 14:45:52 2021 -0400

    Add more logging to run docker login

    Add error log for running docker login. The new log could
    help identify docker login failure.

    Closes-Bug: 1930310
    Change-Id: I8a709fb6665de8301fbe3022563499a92b2a0211
    Signed-off-by: Bin Qian <email address hidden>

commit 31c77439d2cea590dfcca13cfa646522665f8686
Author: albailey <email address hidden>
Date: Fri May 28 13:42:42 2021 -0500

    Fix controller-0 downgrade failing to kill ceph

    kill_ceph_storage_monitor tried to manipulate a pmon
    file that does not exist in an AIO-DX environment.

    We no longer invoke kill_ceph_storage_monitor in an
    AIO SX or DX env.

    This allows: "system host-downgrade controller-0"
    to proceed in an AIO-DX environment where that second
    controller (controller-0) was upgraded.

    Partial-Bug: 1929884
    Signed-off-by: albailey <email address hidden>
    Change-Id: I633853f75317736084feae96b5b849c601204c13

commit 0dc99eee608336fe01b58821ea404286371f1408
Author: albailey <email address hidden>
Date: Fri May 28 11:05:43 2021 -0500

    Fix file permissions failure during duplex upgrade abort

    When issuing a downgrade for controller-0 in a duplex upgrade
    abort and rollback scenario, the downgrade command was failing
    because the sysinv API does not have root permissions to set
    a file flag.
    The fix is to use RPC so the conductor can create the flag
    and allow the downgrade for controller-0 to get further.

    Partial-Bug: 1929884
    Signed-off-by: albailey <email address hidden>
    Change-Id: I913bcad73309fe887a12cbb016a518da93327947

commit 7ef3724dad173754e40b45538b1cc726a458cc1c
Author: Chen, Haochuan Z <email address hidden>
Date: Tue May 25 16:16:29 2021 +0800

    Fix bug rook-ceph provision with multi osd on one host

    Test case:
    1, deploy simplex system
    2, apply rook-ceph with below override value
    value.yaml
    cluster:
      storage:
        nodes:
        - name: controller-0
          devices:
          - name: sdb
          - name: sdc
    3, reboot

    Without this fix, only osd pod could launch successfully after boot
    as vg start with ceph could not correctly add in sysinv-database

    Closes-bug: 1929511

    Change-Id: Ia5be599cd168d13d2aab7b5e5890376c3c8a0019
    Signed-off-by: Chen, Haochuan Z <email address hidden>

commit 23505ba77d76114cf8a0bf833f9a5bcd05bc1dd1
Author: Angie Wang <email address hidden>
Date: Tue May 25 18:49:21 2021 -0400

    Fix issue in partition data migration script

    The created partition dictonary partition_map is not
    an ordered dict so we need to sort it by its key -
    device node when iterating it to adjust the device
    nodes/paths for user created extra partitions to ensure
    the number of device node...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on config (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/config/+/793696

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/config/+/793460

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.