neutron

fullstack security groups test fails because ncat process don't starts

Bug #1744402 reported by Slawek Kaplonski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Incomplete
High
Slawek Kaplonski

Bug Description

Sometimes fullstack test neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork fails because "ncat" process don't starts properly:

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "neutron/tests/base.py", line 132, in func
        return f(self, *args, **kwargs)
      File "neutron/tests/fullstack/test_securitygroup.py", line 163, in test_securitygroup
        net_helpers.NetcatTester.TCP)
      File "neutron/tests/fullstack/test_securitygroup.py", line 68, in assert_connection
        self.assertTrue(netcat.test_connectivity())
      File "neutron/tests/common/net_helpers.py", line 509, in test_connectivity
        self.client_process.writeline(testing_string)
      File "neutron/tests/common/net_helpers.py", line 459, in client_process
        self.establish_connection()
      File "neutron/tests/common/net_helpers.py", line 489, in establish_connection
        address=self.address)
      File "neutron/tests/common/net_helpers.py", line 537, in _spawn_nc_in_namespace
        proc = RootHelperProcess(cmd, namespace=namespace)
      File "neutron/tests/common/net_helpers.py", line 288, in __init__
        self._wait_for_child_process()
      File "neutron/tests/common/net_helpers.py", line 321, in _wait_for_child_process
        "in %d seconds" % (self.cmd, timeout)))
      File "neutron/common/utils.py", line 649, in wait_until_true
        raise exception
    RuntimeError: Process ['ncat', u'20.0.0.5', '3333', '-w', '20'] hasn't been spawned in 20 seconds

Slawek Kaplonski (slaweq)
Changed in neutron:
assignee: nobody → Slawek Kaplonski (slaweq)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/541242

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/541242
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=685596f89c02711410bd6b650a12264ca99145fe
Submitter: Zuul
Branch: master

commit 685596f89c02711410bd6b650a12264ca99145fe
Author: Sławek Kapłoński <email address hidden>
Date: Tue Feb 6 12:19:49 2018 +0100

    [Fullstack] Mark security group test as unstable

    As is described in related bug report this tests is failing
    quite often due to some problem with spawning "ncat" process.

    Change-Id: I0651da07fb786cce82262703bfab8ca30ad4125d
    Related-Bug: 1744402

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

After unmark this test as unstable it starts failing from time to time again. For example: http://logs.openstack.org/34/555834/3/check/neutron-fullstack/b74983d/logs/testr_results.html.gz

So I need to dig more into it

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/556155

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/556155
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1e49eab25080ff82002604a8f38eafbfa9a4f9b9
Submitter: Zuul
Branch: master

commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

Looks that it happens again: http://logs.openstack.org/22/557122/4/check/neutron-fullstack/185813b/logs/testr_results.html.gz
I will investigate there

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

No hits in last few weeks: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%5C%22neutron-fullstack%5C%22%20AND%20message%3A%5C%22hasn't%20been%20spawned%20in%2020%20seconds%5C%22
I will close this bug report now

Changed in neutron:
status: Confirmed → Incomplete
Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
tags: added: neutron-easy-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/573621

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/573624

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/573625

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/573625
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=69cab95e0204c1615f04c6bb4995668edab04f10
Submitter: Zuul
Branch: stable/ocata

commit 69cab95e0204c1615f04c6bb4995668edab04f10
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402
    (cherry picked from commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9)

tags: added: in-stable-ocata
tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/573624
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e8a9cc1ea9d058871ef108b72b7c58e81f9003d8
Submitter: Zuul
Branch: stable/pike

commit e8a9cc1ea9d058871ef108b72b7c58e81f9003d8
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402
    (cherry picked from commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/573621
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=92a103bdbaaab7bb31dcac6711a8364ca38a2397
Submitter: Zuul
Branch: stable/queens

commit 92a103bdbaaab7bb31dcac6711a8364ca38a2397
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402
    (cherry picked from commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/830732

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/830732
Committed: https://opendev.org/openstack/neutron/commit/ba4bfa8842db55463059319138151151f1802575
Submitter: "Zuul (22348)"
Branch: master

commit ba4bfa8842db55463059319138151151f1802575
Author: yatinkarel <email address hidden>
Date: Thu Feb 24 10:42:20 2022 +0530

    [Fullstack] Block until dhcp config is done

    Need to ensure ip is configured in vm before doing other
    netcat connectivity checks. It was missed in initial fix[1].

    [1] https://review.opendev.org/c/openstack/neutron/+/556155

    Related-Bug: #1744402
    Change-Id: Icb2c9c056fa9ab5cf42140e824eee4d1427c8f40

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/c/openstack/neutron/+/830374
Committed: https://opendev.org/openstack/neutron/commit/63a8f964269157c07affce0ff616378d21a0d58c
Submitter: "Zuul (22348)"
Branch: master

commit 63a8f964269157c07affce0ff616378d21a0d58c
Author: yatinkarel <email address hidden>
Date: Tue Feb 22 16:34:26 2022 +0530

    Revert "[Fullstack] Mark security groups test as unstable"

    This reverts commit e748e01656e6d2db1510d85e4c25e21ff394c971.
    The known issue is fixed with [1].

    [1] https://review.opendev.org/c/openstack/neutron/+/830732

    Closes-Bug: #1742401
    Related-Bug: #1744402
    Change-Id: I4f503e89e0aa8e63abcea9c951a973168dcb7148

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.