neutron

[FT] "TestMonitorDaemon._run_monitor" failing radomly, initial message not written

Bug #2035382 reported by Rodolfo Alonso
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
In Progress
High
Miro Tomaska

Bug Description

The method ``TestMonitorDaemon._run_monitor`` is randomly failing. The initial message expected after "keepalived_state_change" process is started is not written in the logs.

Logs: https://8271f43479f81f7d3395-4bdeef087e9d95514555f2932706a956.ssl.cf1.rackcdn.com/893555/1/check/neutron-functional-with-uwsgi/90dde54/testr_results.html

Snippet: https://paste.opendev.org/show/bJCAjIqiUOcWB0xBhPnX/

Miro Tomaska (mtomaska)
tags: added: functional-tests
Miro Tomaska (mtomaska)
Changed in neutron:
importance: Undecided → High
Miro Tomaska (mtomaska)
Changed in neutron:
assignee: nobody → Miro Tomaska (mtomaska)
Revision history for this message
Miro Tomaska (mtomaska) wrote :

It appears this is the same issue we fixed few months ago [1][2]. There we have raised the log file read timeout by 3 seconds, totaling to 18 seconds. The Traceback[3] from this failure states that timeout was set to 15 seconds which makes me believe this is an older branch. If I am reading the job output correctly this run on Xena[4] which would explain the failure.

@Rodolfo Alonso, should I just backport[2] to earlier branches? Thanks!

[1]https://bugs.launchpad.net/neutron/+bug/2003022
[2]https://review.opendev.org/c/openstack/neutron/+/870679
[3]https://paste.opendev.org/show/bJCAjIqiUOcWB0xBhPnX/
[4]https://opendev.org/openstack/neutron/src/commit/067364fce50ca4105b3a98f9838d7fecc8e6baec/neutron/tests/functional/agent/l3/test_keepalived_state_change.py#L99

Changed in neutron:
status: New → In Progress
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hi Miro:

That could work, I didn't check the branch of the logs. But as you said, if the timeout is 15 seconds, most probably these are stable ones. Please propose backports for the stable branches and let's check if that solves the issue.

Regards.

Miro Tomaska (mtomaska)
tags: added: stable
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/zed)

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/895667

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/yoga)

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/895668

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/xena)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/895669

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/895670

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895667
Committed: https://opendev.org/openstack/neutron/commit/e27a4af2b1e07043e7a7a1595b810cd9c8e22238
Submitter: "Zuul (22348)"
Branch: stable/zed

commit e27a4af2b1e07043e7a7a1595b810cd9c8e22238
Author: Miguel Lavalle <email address hidden>
Date: Mon Jan 16 16:41:20 2023 -0600

    Add 3 secs to wait for keepalived state change

    We add 3 seconds to the keepalived state change to avoid timeout
    failures in test_read_queue_change_state.

    Change-Id: Ic54bd6f699b91b53f22c93c972d8def6c99d0eb7
    Closes-Bug: #2035382
    Related-Bug: #2003022
    (cherry picked from commit f28e4165993f0c226bd5af54ad6d8c6bb48bff69)

tags: added: in-stable-zed
tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895668
Committed: https://opendev.org/openstack/neutron/commit/edf33d2217f73b8775c76f3cac2fc75ce00632a1
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit edf33d2217f73b8775c76f3cac2fc75ce00632a1
Author: Miguel Lavalle <email address hidden>
Date: Mon Jan 16 16:41:20 2023 -0600

    Add 3 secs to wait for keepalived state change

    We add 3 seconds to the keepalived state change to avoid timeout
    failures in test_read_queue_change_state.

    Change-Id: Ic54bd6f699b91b53f22c93c972d8def6c99d0eb7
    Closes-Bug: #2035382
    Related-Bug: #2003022
    (cherry picked from commit f28e4165993f0c226bd5af54ad6d8c6bb48bff69)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895669
Committed: https://opendev.org/openstack/neutron/commit/9579ce4c35a1ae35370f9305f8f167ab2d4f4c85
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 9579ce4c35a1ae35370f9305f8f167ab2d4f4c85
Author: Miguel Lavalle <email address hidden>
Date: Mon Jan 16 16:41:20 2023 -0600

    Add 3 secs to wait for keepalived state change

    We add 3 seconds to the keepalived state change to avoid timeout
    failures in test_read_queue_change_state.

    Change-Id: Ic54bd6f699b91b53f22c93c972d8def6c99d0eb7
    Closes-Bug: #2035382
    Related-Bug: #2003022
    (cherry picked from commit f28e4165993f0c226bd5af54ad6d8c6bb48bff69)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895670
Committed: https://opendev.org/openstack/neutron/commit/ba0c6a176590206e6de5a513ed757f1c818cc8af
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit ba0c6a176590206e6de5a513ed757f1c818cc8af
Author: Miguel Lavalle <email address hidden>
Date: Mon Jan 16 16:41:20 2023 -0600

    Add 3 secs to wait for keepalived state change

    We add 3 seconds to the keepalived state change to avoid timeout
    failures in test_read_queue_change_state.

    Change-Id: Ic54bd6f699b91b53f22c93c972d8def6c99d0eb7
    Closes-Bug: #2035382
    Related-Bug: #2003022
    (cherry picked from commit f28e4165993f0c226bd5af54ad6d8c6bb48bff69)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.