Charm Helpers

HAproxy binds to IPv6 even if it's disabled in the OS and as a result fails to start

Bug #1719280 reported by Tytus Kurek
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Charm Helpers
Fix Committed
Medium
Tytus Kurek
OpenStack Base Layer
Fix Released
Medium
Tytus Kurek
OpenStack Base Layer 18.02
charms.openstack
Fix Released
Medium
Tytus Kurek

Bug Description

This issue has been faced when deploying designate charm from openstack-charmers-next repository in the HA mode on Ubuntu Trusty where IPv6 is disabled:

$ juju status designate
Model Controller Cloud/Region Version SLA
default bratislava bratislava 2.2.2 unsupported

App Version Status Scale Charm Store Rev OS Notes
designate 2.0.0 blocked 3 designate local 33 ubuntu
designate-hacluster active 3 hacluster local 31 ubuntu

Unit Workload Agent Machine Public address Ports Message
designate/51* blocked idle 22/lxd/23 100.107.2.1 9001/tcp Ports which should be open, but are not: 9001
  designate-hacluster/96 active executing 100.107.2.1 Unit is ready and clustered
designate/53 blocked idle 23/lxd/23 100.107.66.239 9001/tcp Ports which should be open, but are not: 9001
  designate-hacluster/97* active executing 100.107.66.239 Unit is ready and clustered
designate/54 blocked idle 24/lxd/23 100.107.66.242 9001/tcp Ports which should be open, but are not: 9001
  designate-hacluster/95 active executing 100.107.66.242 Unit is ready and clustered

Machine State DNS Inst id Series AZ Message
22 started 100.107.2.134 q8eh4f trusty FC1 Deployed
22/lxd/23 started 100.107.2.1 juju-64cbea-22-lxd-23 trusty Container started
23 started 100.107.66.126 fbg4ta trusty FC2 Deployed
23/lxd/23 started 100.107.66.239 juju-64cbea-23-lxd-23 trusty Container started
24 started 100.107.66.128 gdpgys trusty FC2 Deployed
24/lxd/23 started 100.107.66.242 juju-64cbea-24-lxd-23 trusty Container started

Relation Provides Consumes Type
cluster designate designate peer
dns-backend designate designate-bind regular
ha designate designate-hacluster subordinate
identity-service designate keystone regular
shared-db designate percona-cluster regular
amqp designate rabbitmq-server-designate regular
hanode designate-hacluster designate-hacluster peer

This is because "/etc/haproxy/haproxy.cfg" file is being written with the following content:

global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    maxconn 20000
    user haproxy
    group haproxy
    spread-checks 0

defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    retries 3
    timeout queue 5000
    timeout connect 5000
    timeout client 30000
    timeout server 30000

listen stats
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /
    stats auth admin:oetycPlNCmEMfFKN2DfcdV0OfPPPliH1

frontend tcp-in_designate-api_admin
    bind *:9001
    bind :::9001
    acl net_100.107.2.1 dst 100.107.2.1/255.255.252.0
    use_backend designate-api_admin_100.107.2.1 if net_100.107.2.1
    default_backend designate-api_admin_100.107.2.1

backend designate-api_admin_100.107.2.1
    balance leastconn
    server designate-51 100.107.2.1:8991 check
    server designate-53 100.107.66.239:8991 check
    server designate-54 100.107.66.242:8991 check

The problem is that "bind :::9001" statement won't work on the systems with IPv6 disabled and thus, will result in haproxy failing to start. It looks like the base openstack layer does not contain a check for this statement in "haproxy.cfg" template file.

Revision history for this message
Tytus Kurek (tkurek) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/507033/

Changed in layer-openstack:
assignee: nobody → Tytus Kurek (tkurek)
Tytus Kurek (tkurek)
tags: added: 4010 cpe-onsite
Revision history for this message
James Page (james-page) wrote :

Looking at the main charmhelpers template, things work exactly the same way there as well.

Revision history for this message
James Page (james-page) wrote :

frontend tcp-in_{{ service }}
    bind *:{{ ports[0] }}
    bind :::{{ ports[0] }}
    {% for frontend in frontends -%}
    acl net_{{ frontend }} dst {{ frontends[frontend]['network'] }}
    use_backend {{ service }}_{{ frontend }} if net_{{ frontend }}
    {% endfor -%}
    default_backend {{ service }}_{{ default_backend }}

Revision history for this message
James Page (james-page) wrote :

Tytus - do you see this issue with non-reactive charms as well?

Changed in layer-openstack:
status: New → Incomplete
importance: Undecided → Medium
Revision history for this message
Tytus Kurek (tkurek) wrote :

It looks like templates for other charms have been modified manually in this environment:

frontend tcp-in_{{ service }}
    bind *:{{ ports[0] }}
    {% if ipv6 -%}
    bind :::{{ ports[0] }}
    {% endif -%}

It looks like it's a general issue.

Revision history for this message
Tytus Kurek (tkurek) wrote :

It is exposed in file "charmhelpers/contrib/openstack/context.py" file:

        if config('prefer-ipv6'):
            ctxt['ipv6'] = True
            ctxt['local_host'] = 'ip6-localhost'
            ctxt['haproxy_host'] = '::'

Revision history for this message
James Page (james-page) wrote :

That approach is not 100% reliable as its possible that network spaces or config provided networks might also be IPv6 subnet configurations so I'm reticent to tie this to the prefer-ipv6 feature (which I'd really like to drop from the charms as it was implemented to support IPv6 deployments in the context of a very old juju version).

Revision history for this message
James Page (james-page) wrote :

It also breaks the dual-stack work done last cycle, which depends on this binding so I'm a little worried there are alot of moving parts in play here that might break.

Revision history for this message
James Page (james-page) wrote :

We could conditionally enable the IPv6 binding IF IPv6 is enabled on the host system - that would probably work in all use-cases.

Revision history for this message
James Page (james-page) wrote :

The ip charmhelper has a is_ipv6_disabled() function - that information could be passed via a context to the templates so that this sort of thing can be conditionally enabled.

Changed in layer-openstack:
status: Incomplete → Triaged
Changed in charm-helpers:
status: New → Triaged
importance: Undecided → Medium
Changed in layer-openstack:
milestone: none → 18.02
Revision history for this message
James Page (james-page) wrote :

Raising charm-helpers target as well - we'll need to make this part of the HAProxy context and template for classic charm support for running with IPv6 disabled.

Changed in charms.openstack:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Tytus Kurek (tkurek) wrote :

Fix for Charm Helpers proposed to the following branch:

lp:~tkurek/charm-helpers/1719280

Changed in charm-helpers:
assignee: nobody → Tytus Kurek (tkurek)
Revision history for this message
James Page (james-page) wrote :

Hi Tytus

Charm Helpers development moved about two weeks ago over to github - please can you raise your changes as a pull requests against:

    https://github.com/juju/charm-helpers

Cheers

Jamers

Revision history for this message
James Page (james-page) wrote :

No - charm-helpers is not part of openstack; you'll need to fork the project on github.com to your own account, create a branch on that repository, make your changes, push back to your repo copy on github.com and then raise a pull request (which will automatically pop up in the WebUI).

Revision history for this message
Tytus Kurek (tkurek) wrote :

Pull request created.

Revision history for this message
Tytus Kurek (tkurek) wrote :

Fix for charms.openstack proposed to branch: master
Review: https://review.openstack.org/#/c/508888/

Changed in charms.openstack:
assignee: nobody → Tytus Kurek (tkurek)
OpenStack Infra (hudson-openstack)
Changed in charms.openstack:
status: Triaged → In Progress
Tytus Kurek (tkurek)
Changed in layer-openstack:
status: Triaged → In Progress
Changed in charm-helpers:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charms.openstack (master)

Reviewed: https://review.openstack.org/508888
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=82ba9595fa05dc6ddccabb3242b2f9c222f74c6e
Submitter: Zuul
Branch: master

commit 82ba9595fa05dc6ddccabb3242b2f9c222f74c6e
Author: Tytus Kurek <email address hidden>
Date: Mon Oct 2 16:06:33 2017 +0200

    Enable IPv6 check in haproxy template file

    This patchset exposes "ipv6_enabled" variable which can be used to
    add IPv6 check to the haproxy template file.

    Change-Id: I1e7329323cbe4ea9d2d79e4196690efa015aa0f4
    Closes-Bug: #1719280

Changed in charms.openstack:
status: In Progress → Fix Released
Tytus Kurek (tkurek)
Changed in charm-helpers:
status: In Progress → Fix Committed
Tytus Kurek (tkurek)
Changed in layer-openstack:
status: In Progress → Fix Committed
Ryan Beisner (1chb1n)
Changed in layer-openstack:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.