haproxy failing on precise due to ipv6 addresses in conf

Bug #1403616 reported by Ryan Beisner
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
charmhelpers (Juju Charms Collection)
Fix Released
High
Edward Hope-Morley
cinder (Juju Charms Collection)
Fix Released
High
Edward Hope-Morley
glance (Juju Charms Collection)
Fix Released
High
Edward Hope-Morley
keystone (Juju Charms Collection)
Fix Released
High
Edward Hope-Morley
neutron-api (Juju Charms Collection)
Fix Released
High
Edward Hope-Morley
nova-cloud-controller (Juju Charms Collection)
Fix Released
High
Edward Hope-Morley
swift-proxy (Juju Charms Collection)
Fix Released
High
Edward Hope-Morley

Bug Description

cinder (/next charm) is not binding to tcp 8776 on precise

## cli fails:
ubuntu@beisner-bastion:~/bzr/openstack-charm-testing⟫ cinder list
ERROR: Unable to establish connection: HTTPConnectionPool(host='10.5.3.84', port=8776): Max retries exceeded with url: /v1/aa14bdc59aea4e0a9368a954e1f4b464/volumes/detail (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

## fyi other cloud cli ok
ubuntu@beisner-bastion:~/bzr/openstack-charm-testing⟫ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

## keystone service-list && keystone endpoint-list:
http://paste.ubuntu.com/9553373/

## cinder unit log
2014-12-17 18:42:37 INFO juju-log identity-service:29: Wrote template /etc/haproxy/haproxy.cfg.
2014-12-17 18:42:37 INFO identity-service-relation-changed Site openstack_https_frontend already disabled
2014-12-17 18:42:37 INFO identity-service-relation-changed cinder-api stop/waiting
2014-12-17 18:42:37 INFO identity-service-relation-changed cinder-api start/running, process 16101
2014-12-17 18:42:37 INFO identity-service-relation-changed cinder-volume stop/waiting
2014-12-17 18:42:37 INFO identity-service-relation-changed cinder-volume start/running, process 16111
2014-12-17 18:42:37 INFO identity-service-relation-changed cinder-scheduler stop/waiting
2014-12-17 18:42:37 INFO identity-service-relation-changed cinder-scheduler start/running, process 16121
2014-12-17 18:42:37 INFO identity-service-relation-changed * Restarting haproxy haproxy
2014-12-17 18:42:37 INFO identity-service-relation-changed [ALERT] 350/184237 (16125) : Starting frontend tcp-in_cinder_api: cannot bind socket
2014-12-17 18:42:37 INFO identity-service-relation-changed ...fail!
Reading package lists... Doneage-backend-relation-joined
Building dependency tree e-backend-relation-joined
Reading state information... Donebackend-relation-joined
2014-12-17 18:42:42 INFO juju-log storage-backend:30: Registered config file: /etc/cinder/api-paste.ini
2014-12-17 18:42:42 INFO juju-log storage-backend:30: Registered config file: /etc/cinder/cinder.conf
2014-12-17 18:42:42 INFO juju-log storage-backend:30: Registered config file: /etc/haproxy/haproxy.cfg
2014-12-17 18:42:42 INFO juju-log storage-backend:30: Registered config file: /etc/apache2/sites-available/openstack_https_frontend
2014-12-17 18:42:42 INFO juju-log storage-backend:30: [cinder] Unknown hook storage-backend-relation-joined - skipping.

## cinder unit net stat
ubuntu@juju-beis0-machine-5:/var/log/cinder$ sudo netstat -taupn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 10883/apache2
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 800/sshd
tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 7868/tgtd
tcp 0 0 0.0.0.0:8766 0.0.0.0:* LISTEN 16578/python
tcp 0 0 10.5.3.84:32987 10.5.3.90:3306 ESTABLISHED 16621/python
tcp 0 240 10.5.3.84:22 10.5.3.80:56387 ESTABLISHED 16902/sshd: ubuntu
tcp 0 0 10.5.3.84:44890 10.5.3.80:17070 ESTABLISHED 4261/jujud
tcp 0 0 10.5.3.84:32986 10.5.3.90:3306 ESTABLISHED 16598/python
tcp 0 0 10.5.3.84:51008 10.5.3.97:5672 ESTABLISHED 16598/python
tcp 0 0 10.5.3.84:44894 10.5.3.80:17070 ESTABLISHED 4310/jujud
tcp 0 0 10.5.3.84:51010 10.5.3.97:5672 ESTABLISHED 16621/python
tcp 0 0 10.5.3.84:44946 10.5.3.80:17070 ESTABLISHED 11031/jujud
tcp 0 0 10.5.3.84:47952 10.5.3.80:6514 ESTABLISHED 11116/rsyslogd
tcp 0 0 10.5.3.84:51011 10.5.3.97:5672 ESTABLISHED 16621/python
tcp 0 0 10.5.3.84:51004 10.5.3.97:5672 ESTABLISHED 16598/python
tcp6 0 0 :::22 :::* LISTEN 800/sshd
tcp6 0 0 :::3260 :::* LISTEN 7868/tgtd
udp 0 0 0.0.0.0:68 0.0.0.0:* 585/dhclient3

## cinder unit versions
ubuntu@juju-beis0-machine-5:/var/log/cinder$ apt-cache policy cinder-api cinder-common python-cinder
cinder-api:
  Installed: 1:2014.1.3-0ubuntu1~cloud0
  Candidate: 1:2014.1.3-0ubuntu1~cloud0
  Version table:
 *** 1:2014.1.3-0ubuntu1~cloud0 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/icehouse/main amd64 Packages
        100 /var/lib/dpkg/status
cinder-common:
  Installed: 1:2014.1.3-0ubuntu1~cloud0
  Candidate: 1:2014.1.3-0ubuntu1~cloud0
  Version table:
 *** 1:2014.1.3-0ubuntu1~cloud0 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/icehouse/main amd64 Packages
        100 /var/lib/dpkg/status
python-cinder:
  Installed: 1:2014.1.3-0ubuntu1~cloud0
  Candidate: 1:2014.1.3-0ubuntu1~cloud0
  Version table:
 *** 1:2014.1.3-0ubuntu1~cloud0 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/icehouse/main amd64 Packages
        100 /var/lib/dpkg/status

ubuntu@juju-beis0-machine-5:/var/log/cinder$ uname -a
Linux juju-beis0-machine-5 3.2.0-74-virtual #109-Ubuntu SMP Tue Dec 9 17:04:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@juju-beis0-machine-5:/var/log/cinder$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise

## connectivity is good
1 ubuntu@beisner-bastion:~/bzr/openstack-charm-testing⟫ ping -c 2 10.5.3.84
PING 10.5.3.84 (10.5.3.84) 56(84) bytes of data.
64 bytes from 10.5.3.84: icmp_seq=1 ttl=64 time=1.16 ms
64 bytes from 10.5.3.84: icmp_seq=2 ttl=64 time=0.385 ms

--- 10.5.3.84 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.385/0.774/1.164/0.390 ms

ubuntu@beisner-bastion:~/bzr/openstack-charm-testing⟫ ssh 10.5.3.84
Warning: Permanently added '10.5.3.84' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 12.04.5 LTS (GNU/Linux 3.2.0-74-virtual x86_64)

Related branches

Revision history for this message
Ryan Beisner (1chb1n) wrote :
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Also fyi, while this juju deploys cleanly, the post-deployment cli checks catch it and fail the deploy test.

Ryan Beisner (1chb1n)
description: updated
Revision history for this message
Billy Olsen (billy-olsen) wrote :

I suspect this is something of a timing issue where haproxy is being restarted where the service stops, then starts prior to the socket being fully closed leading to the inability for haproxy to bind to it.

Revision history for this message
Billy Olsen (billy-olsen) wrote :

Nope, it appears that haproxy reliably fails to start:

2014-12-17 20:33:01 INFO config-changed * Starting haproxy haproxy
2014-12-17 20:33:01 INFO config-changed [ALERT] 350/203301 (10867) : Starting frontend tcp-in_cinder_api: cannot bind socket
2014-12-17 20:33:01 INFO config-changed ...fail!
2014-12-17 20:33:01 INFO config-changed * Starting web server apache2
2014-12-17 20:33:01 INFO config-changed ...done.

2014-12-17 20:35:49 INFO shared-db-relation-changed * Restarting haproxy haproxy
2014-12-17 20:35:49 INFO shared-db-relation-changed [ALERT] 350/203549 (12158) : Starting frontend tcp-in_cinder_api: cannot bind socket
2014-12-17 20:35:49 INFO shared-db-relation-changed ...fail!

2014-12-17 20:36:07 INFO amqp-relation-changed * Restarting haproxy haproxy
2014-12-17 20:36:07 INFO amqp-relation-changed [ALERT] 350/203607 (13617) : Starting frontend tcp-in_cinder_api: cannot bind socket
2014-12-17 20:36:07 INFO amqp-relation-changed ...fail!

2014-12-17 20:37:35 INFO storage-backend-relation-changed * Restarting haproxy haproxy
2014-12-17 20:37:35 INFO storage-backend-relation-changed [ALERT] 350/203735 (15720) : Starting frontend tcp-in_cinder_api: cannot bind socket
2014-12-17 20:37:35 INFO storage-backend-relation-changed ...fail!

2014-12-17 20:37:42 INFO identity-service-relation-changed * Restarting haproxy haproxy
2014-12-17 20:37:42 INFO identity-service-relation-changed [ALERT] 350/203742 (16379) : Starting frontend tcp-in_cinder_api: cannot bind socket
2014-12-17 20:37:42 INFO identity-service-relation-changed ...fail!

Revision history for this message
Billy Olsen (billy-olsen) wrote :

Okay, after some investigation haproxy is not liking both the ipv4 binding and the ipv6 binding for the port and is bailing if both are specified in precise.

Changed in cinder (Juju Charms Collection):
status: New → Confirmed
assignee: nobody → Billy Olsen (billy-olsen)
Changed in cinder (Juju Charms Collection):
assignee: Billy Olsen (billy-olsen) → Edward Hope-Morley (hopem)
Changed in cinder (Juju Charms Collection):
status: Confirmed → In Progress
Changed in charmhelpers (Juju Charms Collection):
status: New → In Progress
importance: Undecided → High
Changed in cinder (Juju Charms Collection):
importance: Undecided → High
Changed in charmhelpers (Juju Charms Collection):
assignee: nobody → Edward Hope-Morley (hopem)
Changed in glance (Juju Charms Collection):
status: New → In Progress
Changed in keystone (Juju Charms Collection):
status: New → In Progress
Changed in nova-cloud-controller (Juju Charms Collection):
status: New → In Progress
Changed in neutron-api (Juju Charms Collection):
status: New → In Progress
Changed in openstack-dashboard (Juju Charms Collection):
status: New → In Progress
Changed in swift-proxy (Juju Charms Collection):
status: New → In Progress
Changed in glance (Juju Charms Collection):
importance: Undecided → High
Changed in keystone (Juju Charms Collection):
importance: Undecided → High
Changed in neutron-api (Juju Charms Collection):
importance: Undecided → High
Changed in nova-cloud-controller (Juju Charms Collection):
importance: Undecided → High
Changed in openstack-dashboard (Juju Charms Collection):
importance: Undecided → High
Changed in swift-proxy (Juju Charms Collection):
importance: Undecided → High
Changed in glance (Juju Charms Collection):
assignee: nobody → Edward Hope-Morley (hopem)
Changed in keystone (Juju Charms Collection):
assignee: nobody → Edward Hope-Morley (hopem)
Changed in neutron-api (Juju Charms Collection):
assignee: nobody → Edward Hope-Morley (hopem)
Changed in nova-cloud-controller (Juju Charms Collection):
assignee: nobody → Edward Hope-Morley (hopem)
Changed in openstack-dashboard (Juju Charms Collection):
assignee: nobody → Edward Hope-Morley (hopem)
Changed in swift-proxy (Juju Charms Collection):
assignee: nobody → Edward Hope-Morley (hopem)
Changed in charmhelpers (Juju Charms Collection):
status: In Progress → Fix Committed
no longer affects: openstack-dashboard (Juju Charms Collection)
summary: - not binding to tcp 8776 on precise
+ haproxy failing on precise due to ipv6 addresses in conf
Revision history for this message
Ryan Beisner (1chb1n) wrote :

precise-icehouse deploy and CLI quick checks succeed with the attached bundle yaml. other deploy target combos still testing. but p-i was the previous fail issue, so this is good!

neutron net-list: ok 0
neutron subnet-list: ok 0
nova network-list: ok 0
nova keypair-list: ok 0
nova secgroup-list: ok 0
glance image-list: ok 0
nova image-list: ok 0
cinder list: ok 0
swift list: ok 0
keystone user-list: ok 0
keystone tenant-list: ok 0
keystone endpoint-list: ok 0
keystone service-list: ok 0
heat stack-list: ok 0

Revision history for this message
Ryan Beisner (1chb1n) wrote :

confirmed a-ok on p-i, t-i, t-j, u-j. thank you, all!

Changed in cinder (Juju Charms Collection):
status: In Progress → Fix Committed
Changed in glance (Juju Charms Collection):
status: In Progress → Fix Committed
Changed in keystone (Juju Charms Collection):
status: In Progress → Fix Committed
Changed in neutron-api (Juju Charms Collection):
status: In Progress → Fix Committed
Changed in nova-cloud-controller (Juju Charms Collection):
status: In Progress → Fix Committed
Changed in swift-proxy (Juju Charms Collection):
status: In Progress → Fix Committed
James Page (james-page)
Changed in cinder (Juju Charms Collection):
milestone: none → 15.01
Changed in charmhelpers (Juju Charms Collection):
milestone: none → 15.01
Changed in glance (Juju Charms Collection):
milestone: none → 15.01
Changed in swift-proxy (Juju Charms Collection):
milestone: none → 15.01
Changed in keystone (Juju Charms Collection):
milestone: none → 15.01
Changed in neutron-api (Juju Charms Collection):
milestone: none → 15.01
Changed in nova-cloud-controller (Juju Charms Collection):
milestone: none → 15.01
James Page (james-page)
Changed in cinder (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in charmhelpers (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in glance (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in swift-proxy (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in keystone (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in neutron-api (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in nova-cloud-controller (Juju Charms Collection):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.