kolla

bluebox gate generates invalid password in keepalived logs

Bug #1568818 reported by Steven Dake
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Critical
Jeffrey Zhang
kolla newton-1 "n1"
Liberty
Fix Released
Critical
Jeffrey Zhang
kolla 1.1.3 "liberty"
Mitaka
Fix Released
Critical
Jeffrey Zhang
kolla 2.0.3 "mitaka"

Bug Description

bluebox gate blocks at waiting for VIP to become available:
http://logs.openstack.org/40/299540/1/check/gate-kolla-dsvm-deploy-centos-source/6c69e15/console.html#_2016-04-11_09_06_37_454

This is presumably because keepalived cannot create the VIP. There is an invalid password error here:
http://logs.openstack.org/40/299540/1/check/gate-kolla-dsvm-deploy-centos-source/6c69e15/logs/kolla/keepalived/keepalived.log

This only happens on bluebox gates.

Steven Dake (sdake)
Changed in kolla:
status: New → Confirmed
importance: Undecided → High
milestone: none → newton-1
Jeffrey Zhang (jeffrey4l)
Changed in kolla:
assignee: nobody → Jeffrey Zhang (jeffrey4l)
OpenStack Infra (hudson-openstack)
Changed in kolla:
status: Confirmed → In Progress
Revision history for this message
Steven Dake (sdake) wrote :

rechecks are suitable until 4/15 release, at which point we should elevate the priority of this gate bug.

Changed in kolla:
importance: High → Low
Revision history for this message
Hui Kang (huikang27) wrote :

not sure if this patch is relevant: https://review.openstack.org/294291

- Hui

Revision history for this message
Jeffrey Zhang (jeffrey4l) wrote :

Hui,

No, see the keepalived log file

```
<6>2016-04-11T09:06:54Z node1 Keepalived_vrrp[22]: VRRP_Instance(kolla_internal_vip_51) ignoring received advertisment...
<6>2016-04-11T09:06:55Z node1 Keepalived_vrrp[22]: receive an invalid passwd!
<6>2016-04-11T09:06:55Z node1 Keepalived_vrrp[22]: bogus VRRP packet received on eth0 !!!
<6>2016-04-11T09:06:55Z node1 Keepalived_vrrp[22]: VRRP_Instance(kolla_internal_vip_51) ignoring received advertisment...
<6>2016-04-11T09:06:56Z node1 Keepalived_vrrp[22]: receive an invalid passwd!
<6>2016-04-11T09:06:56Z node1 Keepalived_vrrp[22]: bogus VRRP packet received on eth0 !!!
<6>2016-04-11T09:06:56Z node1 Keepalived_vrrp[22]: VRRP_Instance(kolla_internal_vip_51) ignoring received advertisment...
<6>2016-04-11T09:06:57Z node1 Keepalived_vrrp[22]: receive an invalid passwd!
<6>2016-04-11T09:06:57Z node1 Keepalived_vrrp[22]: bogus VRRP packet received on eth0 !!!
<6>2016-04-11T09:06:57Z node1 Keepalived_vrrp[22]: VRRP_Instance(kolla_internal_vip_51) ignoring received advertisment...
<6>2016-04-11T09:06:58Z node1 Keepalived_vrrp[22]: receive an invalid passwd!
```

if is caused by the another keepavlied instance exist with different auth.

Revision history for this message
Steven Dake (sdake) wrote :

note openstack-infra said their firewall rules would prevent leaking of multicast packets. Worth verifying if that is the case and reporting back here.

Revision history for this message
Jeffrey Zhang (jeffrey4l) wrote :

https://review.openstack.org/304080

Fix is here

Steven Dake (sdake)
Changed in kolla:
importance: Low → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/304080
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=f93ed9a935c9e9b159cb42c548f4d6d3383f36f2
Submitter: Jenkins
Branch: master

commit f93ed9a935c9e9b159cb42c548f4d6d3383f36f2
Author: Jeffrey Zhang <jeffrey.zhang@99cloud.net>
Date: Mon Apr 11 20:11:33 2016 +0800

    Use a random virtual_router_id in the keepalived

    Use a random virtual_router_id in case there are multi keepalived
    instances exist at the same time in the gate.

    Closes-Bug: #1568818
    Change-Id: Iebb3dec088a21820e641b63bb2a5f40586d1df8c

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/kolla 3.0.0.0b1

This issue was fixed in the openstack/kolla 3.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/357997

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/357998

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/liberty)

Reviewed: https://review.openstack.org/357998
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=c600eb6ee994ac66fdaa94332e167866b599e271
Submitter: Jenkins
Branch: stable/liberty

commit c600eb6ee994ac66fdaa94332e167866b599e271
Author: Jeffrey Zhang <jeffrey.zhang@99cloud.net>
Date: Mon Apr 11 20:11:33 2016 +0800

    Use a random virtual_router_id in the keepalived

    Use a random virtual_router_id in case there are multi keepalived
    instances exist at the same time in the gate.

    Closes-Bug: #1568818
    Change-Id: Iebb3dec088a21820e641b63bb2a5f40586d1df8c
    (cherry picked from commit f93ed9a935c9e9b159cb42c548f4d6d3383f36f2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/mitaka)

Reviewed: https://review.openstack.org/357997
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=c07c8b4cfada3345ab33d638e202b9272fa8f5a6
Submitter: Jenkins
Branch: stable/mitaka

commit c07c8b4cfada3345ab33d638e202b9272fa8f5a6
Author: Jeffrey Zhang <jeffrey.zhang@99cloud.net>
Date: Mon Apr 11 20:11:33 2016 +0800

    Use a random virtual_router_id in the keepalived

    Use a random virtual_router_id in case there are multi keepalived
    instances exist at the same time in the gate.

    Closes-Bug: #1568818
    Change-Id: Iebb3dec088a21820e641b63bb2a5f40586d1df8c
    (cherry picked from commit f93ed9a935c9e9b159cb42c548f4d6d3383f36f2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/359450

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 2.0.3

This issue was fixed in the openstack/kolla 2.0.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla (master)

Change abandoned by Christian Berendt (<email address hidden>) on branch: master
Review: https://review.openstack.org/359450

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.