kolla

Logger resource temporarily unavailable alert in HAProxy logs

Bug #1549753 reported by Éric Lemoine
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Medium
Vikram Hosakote
kolla mitaka-rc1 "mitaka-rc1"

Bug Description

There are alert log messages in the HAProxy logs:

    [ALERT] 055/110341 (16) : sendto logger #1 failed: Resource temporarily unavailable (errno=11)

This is related to the log socket not being available when the haproxy process starts. The "wait for log socket" code in docker/base/start.sh is supposed to protect us from that, but it looks like that does not work as expected.

Steven Dake (sdake)
Changed in kolla:
status: New → Triaged
status: Triaged → Confirmed
Steven Dake (sdake)
Changed in kolla:
importance: Medium → High
milestone: mitaka-3 → mitaka-rc1
Revision history for this message
Éric Lemoine (elemoine) wrote :

Contrary to what's stated in the Bug Description, the problem is not related to the log socket not being available when the haproxy process starts.

The problem is related to net.unix.max_dgram_qlen being too small (10) by default. By increasing net.unix.max_dgram_qlen to 100 (for example) the problem goes away.

See https://bugzilla.redhat.com/show_bug.cgi?id=1257690 for example.

I am going to change the Importance from High to Medium, as the consequence of this issue is just the loss a few log entries at HAProxy startup time.

Changed in kolla:
importance: High → Medium
Revision history for this message
Vikram Hosakote (vhosakot) wrote :

This bug may raise false alarm about haproxy to the operator and monitoring tools even though haproxy is running fine and there is no issue.

I will fix this bug by increasing the datagram receive queue length of haproxy's socket.

Changed in kolla:
status: Confirmed → In Progress
assignee: nobody → Vikram Hosakote (vhosakot)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.openstack.org/292075

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/292075
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=9d1911c5a497fe625989f5795676d9f2fa4029e6
Submitter: Jenkins
Branch: master

commit 9d1911c5a497fe625989f5795676d9f2fa4029e6
Author: Vikram Hosakote <email address hidden>
Date: Sun Mar 13 06:47:38 2016 +0000

    Fix false alert messages in HAProxy logs

    Bug #1549753 raises false alarm about haproxy to the operator and
    monitoring tools even though haproxy is running fine and there is
    no issue.

    haproxy uses UDP datagram socket for logging and the default queue
    length of 10 datagrams causes false alert messages in haproxy
    logs. This patch set fixes this bug by increasing the queue length
    to 128.

    Test results of the fix in this patch set are at
    http://paste.openstack.org/show/490302/.

    Change-Id: I64fdd17cd278d51616aa0dfa4e8d2a422c8ee630
    Closes-Bug: #1549753

Changed in kolla:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.