tripleo

network stop working after reboot overcloud nodes on tripleo-quickstart deployments

Bug #1932143 reported by Juan Badia Payno
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Medium
Unassigned

Bug Description

Deployed the environment with tripleo-quickstart

Starting console:
.....
....
[ OK ] Started Login Service.
[ 9.147419] cloud-init[953]: Cloud-init v. 20.3-10.el8_4.3 running 'init-local' at Wed, 16 Jun 2021 10:04:57 +0000. Up 8.64 seconds.
[ OK ] Started Initial cloud-init job (pre-networking).
[ OK ] Reached target Network (Pre).
         Starting Open vSwitch Database Unit...
[ OK ] Started Open vSwitch Database Unit.
         Starting Open vSwitch Delete Transient Ports...
[ OK ] Started Open vSwitch Delete Transient Ports.
         Starting Open vSwitch Forwarding Unit...
[ 10.744922] openvswitch: Open vSwitch switching datapath
[ 10.814128] device ovs-system entered promiscuous mode
[ 10.816412] Timeout policy base is empty
[ 10.817097] Failed to associated timeout policy `ovs_test_tp'
[ 10.846429] device br-int entered promiscuous mode
[ 10.864947] device vlan20 entered promiscuous mode
[ 10.877688] device vlan10 entered promiscuous mode
[ 10.892602] device vlan40 entered promiscuous mode
[ 10.919643] device br-ex entered promiscuous mode
[ 10.936645] device vlan50 entered promiscuous mode
[ 10.953824] device vlan30 entered promiscuous mode
[ 10.954891] device ens3 entered promiscuous mode
[ OK ] Started Open vSwitch Forwarding Unit.
         Starting Open vSwitch...
[ OK ] Started Open vSwitch.
         Starting Network Manager...
[ OK ] Started Network Manager.
         Starting Network Manager Wait Online...
         Starting Hostname Service...
[ OK ] Started Hostname Service.
         Starting Network Manager Script Dispatcher Servi[ 11.246110] IPv6: ADDRCONF(NETDEV_UP): br-int: link is not ready

[ OK ] Started Network Manager Script Dispatcher Service.
[ OK ] Started Network Manager Wait Online.
         Starting LSB: Bring up/down networking...
[** ] A start job is running for LSB: Bri…up/down networking (17s / 5min 4s)[ 23.973513] IPv4: martian source 172.16.2.154 from 172.16.2.25, on dev vlan20
[ 23.978021] ll header: 00000000: ff ff ff ff ff ff 9e 0a 03 f8 3b 32 08 06 ..........;2..
[* ] A start job is running for LSB: Bri…up/down networking (17s / 5min 4s)[ 24.522446] IPv4: martian source 172.16.2.154 from 172.16.2.141, on dev vlan20
[ 24.523794] ll header: 00000000: ff ff ff ff ff ff 66 3d 12 9c 46 fe 08 06 ......f=..F...
[** ] A start job is running for LSB: Bri…up/down networking (18s / 5min 4s)[ 24.997535] IPv4: martian source 172.16.2.154 from 172.16.2.25, on dev vlan20
[ 24.998919] ll header: 00000000: ff ff ff ff ff ff 9e 0a 03 f8 3b 32 08 06 ..........;2..
[*** ] A start job is running for LSB: Bri…up/down networking (18s / 5min 4s)[ 25.546878] IPv4: martian source 172.16.2.154 from 172.16.2.141, on dev vlan20
[ 25.548312] ll header: 00000000: ff ff ff ff ff ff 66 3d 12 9c 46 fe 08 06 ......f=..F...
[ *** ] A start job is running for LSB: Bri…up/down networking (19s / 5min 4s)[ 26.021439] IPv4: martian source 172.16.2.154 from 172.16.2.25, on dev vlan20
[ 26.022901] ll header: 00000000: ff ff ff ff ff ff 9e 0a 03 f8 3b 32 08 06 ..........;2..
[ *** ] A start job is running for LSB: Bri…up/down networking (20s / 5min 4s)[ 26.570483] IPv4: martian source 172.16.2.154 from 172.16.2.141, on dev vlan20
[ 26.571978] ll header: 00000000: ff ff ff ff ff ff 66 3d 12 9c 46 fe 08 06 ......f=..F...
[ 27.045354] IPv4: martian source 172.16.2.154 from 172.16.2.25, on dev vlan20
[ 27.047131] ll header: 00000000: ff ff ff ff ff ff 9e 0a 03 f8 3b 32 08 06 ..........;2..
[ **] A start job is running for LSB: Bri…up/down networking (21s / 5min 4s)[ 28.070289] IPv4: martian source 172.16.1.133 from 172.16.1.222, on dev vlan30
[ 28.072105] ll header: 00000000: ff ff ff ff ff ff 8e 5d 4a 9a 63 1a 08 06 .......]J.c...
[ *] A start job is running for LSB: Bri…up/down networking (21s / 5min 4s)[ 28.298578] IPv4: martian source 172.16.1.133 from 172.16.1.79, on dev vlan30
[ 28.300366] ll header: 00000000: ff ff ff ff ff ff 4a be 15 e8 f6 2f 08 06 ......J..../..
[ **] A start job is running for LSB: Bri…up/down networking (22s / 5min 4s)[ 29.093469] IPv4: martian source 172.16.1.133 from 172.16.1.222, on dev vlan30
[ 29.095082] ll header: 00000000: ff ff ff ff ff ff 8e 5d 4a 9a 63 1a 08 06 .......]J.c...
[ 29.323291] IPv4: martian source 172.16.1.133 from 172.16.1.79, on dev vlan30
[ 29.324716] ll header: 00000000: ff ff ff ff ff ff 4a be 15 e8 f6 2f 08 06 ......J..../..
[ *** ] A start job is running for LSB: Bri…up/down networking (23s / 5min 4s)[ 30.117440] IPv4: martian source 172.16.1.133 from 172.16.1.222, on dev vlan30
[ 30.119352] ll header: 00000000: ff ff ff ff ff ff 8e 5d 4a 9a 63 1a 08 06 .......]J.c...
[ *** ] A start job is running for LSB: Bri…up/down networking (24s / 5min 4s)[ 31.141234] IPv4: martian source 172.16.1.133 from 172.16.1.222, on dev vlan30
[ 31.142676] ll header: 00000000: ff ff ff ff ff ff 8e 5d 4a 9a 63 1a 08 06 .......]J.c...
[*** ] A start job is running for LSB: Bri…up/down networking (24s / 5min 4s)[ 31.254367] IPv4: martian source 172.16.1.133 from 172.16.1.79, on dev vlan30
[ 31.255779] ll header: 00000000: ff ff ff ff ff ff 4a be 15 e8 f6 2f 08 06 ......J..../..
[ OK ] Started LSB: Bring up/down networking.
         Starting Initial cloud-init job (metadata service crawler)...
[ OK ] Reached target Network.
         Starting Dynamic System Tuning Daemon...
         Starting Enable periodic update of entitlement certificates....
         Starting Dynamic Login...
         Starting Neutron cleanup on startup...
         Starting GSSAPI Proxy Daemon...
[ OK ] Started Enable periodic update of entitlement certificates..
[ OK ] Started GSSAPI Proxy Daemon.
[ OK ] Started Dynamic Login.
[ OK ] Reached target NFS client services.
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
[ OK ] Started Neutron cleanup on startup.
[ OK ] Started Dynamic System Tuning Daemon.
[ 40.984742] cloud-init[2163]: Cloud-init v. 20.3-10.el8_4.3 running 'init' at Wed, 16 Jun 2021 10:05:29 +0000. Up 40.83 seconds.
[ 40.986900] cloud-init[2163]: ci-info: +++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++++
[ 40.989063] cloud-init[2163]: ci-info: +------------+-------+------------------------------+---------------+--------+-------------------+
[ 40.991107] cloud-init[2163]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 40.993225] cloud-init[2163]: ci-info: +------------+-------+------------------------------+---------------+--------+-------------------+
[ 40.995345] cloud-init[2163]: ci-info: | br-ex | True | 192.168.24.12 | 255.255.255.0 | global | 00:80:64:5e:09:bd |
[ 40.997354] cloud-init[2163]: ci-info: | br-ex | True | fe80::280:64ff:fe5e:9bd/64 | . | link | 00:80:64:5e:09:bd |
[ 40.999421] cloud-init[2163]: ci-info: | br-int | True | . | . | . | 3e:17:6c:8c:59:9f |
[ 41.001525] cloud-init[2163]: ci-info: | ens3 | True | fe80::280:64ff:fe5e:9bd/64 | . | link | 00:80:64:5e:09:bd |
[ 41.003576] cloud-init[2163]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
[ 41.005625] cloud-init[2163]: ci-info: | lo | True | ::1/128 | . | host | . |
[ 41.007652] cloud-init[2163]: ci-info: | ovs-system | False | . | . | . | 4a:55:55:c4:fb:74 |
[ 41.009682] cloud-init[2163]: ci-info: | vlan10 | True | 10.0.0.241 | 255.255.255.0 | global | 0a:fa:34:61:6d:62 |
[ 41.011732] cloud-init[2163]: ci-info: | vlan10 | True | fe80::8fa:34ff:fe61:6d62/64 | . | link | 0a:fa:34:61:6d:62 |
[ 41.013695] cloud-init[2163]: ci-info: | vlan20 | True | 172.16.2.154 | 255.255.255.0 | global | f6:09:96:6a:13:72 |
[ 41.015795] cloud-init[2163]: ci-info: | vlan20 | True | fe80::f409:96ff:fe6a:1372/64 | . | link | f6:09:96:6a:13:72 |
[ 41.017717] cloud-init[2163]: ci-info: | vlan30 | True | 172.16.1.133 | 255.255.255.0 | global | 12:5a:c5:ed:c3:6a |
[ 41.019827] cloud-init[2163]: ci-info: | vlan30 | True | fe80::105a:c5ff:feed:c36a/64 | . | link | 12:5a:c5:ed:c3:6a |
[ 41.022118] cloud-init[2163]: ci-info: | vlan40 | True | 172.16.3.169 | 255.255.255.0 | global | 52:83:45:81:63:f8 |
[ 41.024319] cloud-init[2163]: ci-info: | vlan40 | True | fe80::5083:45ff:fe81:63f8/64 | . | link | 52:83:45:81:63:f8 |
[ 41.026534] cloud-init[2163]: ci-info: | vlan50 | True | 172.16.0.240 | 255.255.255.0 | global | 9e:26:80:e5:bb:dd |
[ 41.028691] cloud-init[2163]: ci-info: | vlan50 | True | fe80::9c26:80ff:fee5:bbdd/64 | . | link | 9e:26:80:e5:bb:dd |
[ 41.030714] cloud-init[2163]: ci-info: +------------+-------+------------------------------+---------------+--------+-------------------+
[ 41.032757] cloud-init[2163]: ci-info: ++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++
[ 41.034462] cloud-init[2163]: ci-info: +-------+--------------+----------+---------------+-----------+-------+
[ 41.036208] cloud-init[2163]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
[ 41.037892] cloud-init[2163]: ci-info: +-------+--------------+----------+---------------+-----------+-------+
[ 41.039552] cloud-init[2163]: ci-info: | 0 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | vlan10 | UG |
[ 41.041102] cloud-init[2163]: ci-info: | 1 | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | vlan10 | U |
[ 41.042725] cloud-init[2163]: ci-info: | 2 | 172.16.0.0 | 0.0.0.0 | 255.255.255.0 | vlan50 | U |
[ 41.044329] cloud-init[2163]: ci-info: | 3 | 172.16.1.0 | 0.0.0.0 | 255.255.255.0 | vlan30 | U |
[ 41.045977] cloud-init[2163]: ci-info: | 4 | 172.16.2.0 | 0.0.0.0 | 255.255.255.0 | vlan20 | U |
[ 41.047534] cloud-init[2163]: ci-info: | 5 | 172.16.3.0 | 0.0.0.0 | 255.255.255.0 | vlan40 | U |
[ 41.049107] cloud-init[2163]: ci-info: | 6 | 192.168.24.0 | 0.0.0.0 | 255.255.255.0 | br-ex | U |
[ 41.052331] cloud-init[2163]: ci-info: +-------+--------------+----------+---------------+-----------+-------+
[ 41.053822] cloud-init[2163]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[ 41.055075] cloud-init[2163]: ci-info: +-------+-------------+---------+-----------+-------+
[ 41.056268] cloud-init[2163]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[ 41.057468] cloud-init[2163]: ci-info: +-------+-------------+---------+-----------+-------+
[ 41.058561] cloud-init[2163]: ci-info: | 10 | fe80::/64 | :: | br-ex | U |
[ 41.060245] cloud-init[2163]: ci-info: | 11 | fe80::/64 | :: | ens3 | U |
[ 41.061526] cloud-init[2163]: ci-info: | 12 | fe80::/64 | :: | vlan10 | U |
[ 41.062667] cloud-init[2163]: ci-info: | 13 | fe80::/64 | :: | vlan20 | U |
[ 41.063861] cloud-init[2163]: ci-info: | 14 | fe80::/64 | :: | vlan30 | U |
[ 41.065076] cloud-init[2163]: ci-info: | 15 | fe80::/64 | :: | vlan40 | U |
[ 41.066328] cloud-init[2163]: ci-info: | 16 | fe80::/64 | :: | vlan50 | U |
[ 41.067536] cloud-init[2163]: ci-info: | 18 | local | :: | br-ex | U |
[ 41.068745] cloud-init[2163]: ci-info: | 19 | local | :: | ens3 | U |
[ 41.069949] cloud-init[2163]: ci-info: | 20 | local | :: | vlan10 | U |
[ 41.071180] cloud-init[2163]: ci-info: | 21 | local | :: | vlan30 | U |
[ 41.072259] cloud-init[2163]: ci-info: | 22 | local | :: | vlan40 | U |
[ 41.073335] cloud-init[2163]: ci-info: | 23 | local | :: | vlan50 | U |
[ 41.074431] cloud-init[2163]: ci-info: | 24 | local | :: | vlan20 | U |
[ 41.075525] cloud-init[2163]: ci-info: | 25 | multicast | :: | br-int | U |
[ 41.076626] cloud-init[2163]: ci-info: | 26 | multicast | :: | br-ex | U |
[ 41.077711] cloud-init[2163]: ci-info: | 27 | multicast | :: | ens3 | U |
[ 41.078911] cloud-init[2163]: ci-info: | 28 | multicast | :: | vlan10 | U |
[ 41.080127] cloud-init[2163]: ci-info: | 29 | multicast | :: | vlan20 | U |
[ 41.081339] cloud-init[2163]: ci-info: | 30 | multicast | :: | vlan30 | U |
[ 41.082550] cloud-init[2163]: ci-info: | 31 | multicast | :: | vlan40 | U |
[ 41.083767] cloud-init[2163]: ci-info: | 32 | multicast | :: | vlan50 | U |
[ 41.084959] cloud-init[2163]: ci-info: +-------+-------------+---------+-----------+-------+
[ 41.086048] cloud-init[2163]: 2021-06-16 10:05:30,014 - stages.py[WARNING]: Failed to rename devices: [unknown] Error performing rename('ens3', 'br-ex') for 00:80:64:5e:09:bd, br-ex: Unexpected error while running command.
[ 41.088657] cloud-init[2163]: Command: ['ip', 'link', 'set', 'ens3', 'name', 'br-ex']
[ 41.089654] cloud-init[2163]: Exit code: 2
[ 41.090217] cloud-init[2163]: Reason: -
[ 41.090742] cloud-init[2163]: Stdout:
[ 41.091250] cloud-init[2163]: Stderr: RTNETLINK answers: File exists
[ OK ] Started Initial cloud-init job (metadata service crawler).

Everything seems ok:
[root@overcloud-controller-0 ~]# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master ovs-system state DOWN group default qlen 1000
    link/ether 00:80:64:5e:09:bd brd ff:ff:ff:ff:ff:ff
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 4a:55:55:c4:fb:74 brd ff:ff:ff:ff:ff:ff
4: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 3e:17:6c:8c:59:9f brd ff:ff:ff:ff:ff:ff
5: vlan20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether f6:09:96:6a:13:72 brd ff:ff:ff:ff:ff:ff
    inet 172.16.2.154/24 brd 172.16.2.255 scope global vlan20
       valid_lft forever preferred_lft forever
    inet6 fe80::f409:96ff:fe6a:1372/64 scope link
       valid_lft forever preferred_lft forever
6: vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 0a:fa:34:61:6d:62 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.241/24 brd 10.0.0.255 scope global vlan10
       valid_lft forever preferred_lft forever
    inet6 fe80::8fa:34ff:fe61:6d62/64 scope link
       valid_lft forever preferred_lft forever
7: vlan40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:83:45:81:63:f8 brd ff:ff:ff:ff:ff:ff
    inet 172.16.3.169/24 brd 172.16.3.255 scope global vlan40
       valid_lft forever preferred_lft forever
    inet6 fe80::5083:45ff:fe81:63f8/64 scope link
       valid_lft forever preferred_lft forever
8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:80:64:5e:09:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.24.12/24 brd 192.168.24.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet6 fe80::280:64ff:fe5e:9bd/64 scope link
       valid_lft forever preferred_lft forever
9: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 9e:26:80:e5:bb:dd brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.240/24 brd 172.16.0.255 scope global vlan50
       valid_lft forever preferred_lft forever
    inet6 fe80::9c26:80ff:fee5:bbdd/64 scope link
       valid_lft forever preferred_lft forever
10: vlan30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 12:5a:c5:ed:c3:6a brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.133/24 brd 172.16.1.255 scope global vlan30
       valid_lft forever preferred_lft forever
    inet6 fe80::105a:c5ff:feed:c36a/64 scope link
       valid_lft forever preferred_lft forever

[root@overcloud-controller-0 ~]# ovs-vsctl show
64ecb89c-ab8f-4337-be66-6fff637d8801
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        fail_mode: standalone
        Port vlan30
            tag: 30
            Interface vlan30
                type: internal
        Port vlan20
            tag: 20
            Interface vlan20
                type: internal
        Port vlan50
            tag: 50
            Interface vlan50
                type: internal
        Port ens3
            Interface ens3
        Port vlan40
            tag: 40
            Interface vlan40
                type: internal
        Port vlan10
            tag: 10
            Interface vlan10
                type: internal
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.13.4"
[root@overcloud-controller-0 ~]#

WORKAROUND:
 login into the server and rise the device
 - ifconfig ens3 up

Revision history for this message
Juan Badia Payno (jbadiapa) wrote :

[root@overcloud-controller-0 ~]# dnf history info 16
Transaction ID : 16
Begin time : Sun 13 Jun 2021 04:45:15 PM UTC
Begin rpmdb : 961:a8b8e539792aec96fac1442e4e18fd16b78d2d96
End time : Sun 13 Jun 2021 04:45:16 PM UTC (1 seconds)
End rpmdb : 978:58055238ae455e01a718124030b91f16b57f6671
User : <heat-admin>
Return-Code : Success
Releasever : 8
Command Line : -v -y install python3-psutil python3-debtcollector sos device-mapper-multipath openstack-heat-agents os-net-config jq python3-dbus
Comment :
Packages Altered:
    Install openstack-heat-agents-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install os-net-config-14.2.0-0.20210517154759.352227b.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-ansible-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-apply-config-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-docker-cmd-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-hiera-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-json-file-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-puppet-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install jq-1.5-12.el8.x86_64 @appstream
    Install nispor-1.1.0-1.el8.x86_64 @appstream
    Install nmstate-1.1.0-0.1.el8.noarch @appstream
    Install oniguruma-6.8.2-2.el8.x86_64 @appstream
    Install python3-libnmstate-1.1.0-0.1.el8.noarch @appstream
    Install python3-nispor-1.1.0-1.el8.noarch @appstream
    Install NetworkManager-config-server-1:1.32.0-0.4.el8.noarch @baseos
    Install NetworkManager-ovs-1:1.32.0-0.4.el8.x86_64 @baseos
    Install python3-varlink-29.0.0-1.el8.noarch @baseos

Revision history for this message
Juan Badia Payno (jbadiapa) wrote :

the repo:
---------
[delorean-component-tripleo]
name=delorean-tripleo-operator-ansible-c1527ca9fcd5d2a50af896e2335b0af992d7a94b
baseurl=https://trunk.rdoproject.org/centos8/component/tripleo/c1/52/c1527ca9fcd5d2a50af896e2335b0af992d7a94b_2ecfbcb5
enabled=1
gpgcheck=0
priority=20

Bogdan Dobrelya (bogdando)
Changed in tripleo:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Cristian Le (lecris) wrote :

Recently affected by this issue on normal tripleo deployment. `br-int` and related devices are shown down after a reboot.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.