Mysql backend readiness checks should rely on TCP connectivity instead of sockets

Raised to critical due to many duplicates (i.e. bugs with neutron net-list failures or glance wasn't properly installed failures could be related to this one)

Bogdan, I have not seen this bug reproduced anywhere. Could you add logs or duplicate bugs, please?

here is a source (related) bug for this issue https://bugs.launchpad.net/fuel/+bug/1391213

I don't see how this is a Critical in 5.1.1. Do we have a regression that makes Fuel unable to deploy HA environments?

Here is a new type with TCP checks support https://github.com/dmitryilyin/haproxy_backend_online/

Colleagues,
I doubt that it is High priority issue for stable/5.1.
The reason for it is that I do not see direct connection of this bug with real user experience.
Bogdan, bug you've mentioned as possible source/related one, has comments that these bugs are unrelated. So we need some more clarity and details here.

I'm not only worried that we just need to squash bugs as fast as we can to achieve HCF criteria, but also that first fix for this bug might bear one or more possible bugs which would be hard to catch. So it can be just risky to get a fix of this into maintenance branch.

Moved to 6.1 due to medium priority and SCF

wait-for-haproxy-mysql-backend is important for first node only. Actually it's wrapped in is_primary_controller. Once first node is deployed. It's not so important as HAProxy on every instance verifies MySQL using cluster_check script. We were not able to reproduce this bug, so I think we may close it.

When this issue had been reproduced, the mysqld.log contained "[Note] /usr/sbin/mysqld: ready for connections" message with a *later* timestamp than puppet-apply.log contained one for "Exec[wait-for-haproxy-mysql-backend]) Evaluated". That resulted in Openstack services considered wait-for-haproxy-mysql-backend precondition had been met and started to issue connections to mysql which obviously failed due to it wasn't ready for connections yet.

This bug is floating, hence barely reproducible w/o long run testing for looped deployments

I just put here this reproduced case rom bug #1391213 http://paste.openstack.org/show/134709/ to not search around

Raised to high due to more duplicates incoming

This bug looks inactive, I return it back to Fuel library team

idv1985, I think that it worth to implement your new type here and add it to our manifest.

Thank you for clarification.

So, here is the patch https://review.openstack.org/#/c/136446/

But we should add a check for mysql as well (see original wait-for-haproxy-mysql-backend exec) but modify it in a wat that the mysql check being executed via TCP connect instead of UNIX sockets

Reviewed: https://review.openstack.org/136446
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=21c4055364dedcddbe19980ec01a2efb2d80f10c
Submitter: Jenkins
Branch: master

commit 21c4055364dedcddbe19980ec01a2efb2d80f10c
Author: Dmitry Ilyin <email address hidden>
Date: Tue Jan 13 19:29:01 2015 +0300

    Replace wait-for-haproxy-backends with puppet type

    * Implement a special puppet type to wait until haproxy backend is online.
    * Wait for mysqld backend as a requirement for nova and keystone backends
    * Switch to TCP connect for cluster check instead of UNIX socket:
      - the host to connect is the node's management IP
      - the port is the mysql HAProxy backend port, 3307
      - the user is clustercheck and its password is the same as wsrep user has
      - the timeout is 10 seconds
    * Parametrise galera cluster check class and script
    * Move galera::cluster check to opesntack module (otherwise, there are
      4 more classes to drag all the cluster check parameters through,
      including puppet-mysql module which does not have these params in upstream)
    * Fix docs for cluster check class parameters

    Related-Blueprint: pacemaker-improvements
    Closes-bug: #1394137

    Change-Id: I4eab6af7257270bf3cb2b40a34bcb21c952e8989

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/172843

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/172845

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/5.1
Review: https://review.openstack.org/172845

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/6.0
Review: https://review.openstack.org/172843

The sustaining team should backport this

Won't Fix for 5.1.1-updates and 6.0-updates as we don't expect new 5.1.1 and 6.0 deployments