[systests] deploy_bonding_ha_balance_slb failed, there is no installed cirros-testvm on the controllers

Puppet installs cirros-testvm package on primary controller, Astute also runs image upload on the primary controller, which is node-1, so cirros-testvm package is not a problem. The problem is:

04:35:36 Error communicating with http://10.108.61.2:9292 [Errno 32] Broken pipe

Here's appropriate records from Haproxy log on node-1:

2014-11-11T04:35:36.545182+00:00 info: 10.108.61.3:37007 [11/Nov/2014:04:35:33.719] glance-api glance-api/node-4 0/0/0/-1/2829 502 204 - - SH-- 68/0/0/0/0 0/0 "POST /v1/images HTTP/1.1"

As we can see Haproxy got "502 Bad Gateway" error. Here is a part of glance-api.log from the problem server that was handling this request (node-4):

2014-11-11T04:35:38.020647+00:00 info: 2014-11-11 04:35:38.020 15642 INFO glance.wsgi.server [3f687a27-501c-4a53-96ba-07c81bc6a8eb f419725e6811497dbcf3550718c0e14c b813afb40d424bec945107c488903d75 - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 405, in handle_one_response
    write(''.join(towrite))
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 349, in write
    _writelines(towrite)
  File "/usr/lib/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 32] Broken pipe

@Aleksandr, there was also an error associated with the same req-id before:
2014-11-11T04:35:37.637217+00:00 err: 2014-11-11 04:35:37.635 15642 ERROR swiftclient [3f687a27-501c-4a53-96ba-07c81bc6a8eb f419725e6811497dbcf3550718c0e14c b813afb40d424bec945107c48890
3d75 - - -] Container HEAD failed: http://10.108.61.2:8080/v1/AUTH_750fa318e28c4448b32628f860e9d0f9/glance 404 Not Found
2014-11-11 04:35:37.635 15642 TRACE swiftclient Traceback (most recent call last):
2014-11-11 04:35:37.635 15642 TRACE swiftclient File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1110, in _retry
2014-11-11 04:35:37.635 15642 TRACE swiftclient rv = func(self.url, self.token, *args, **kwargs)
2014-11-11 04:35:37.635 15642 TRACE swiftclient File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 581, in head_container
2014-11-11 04:35:37.635 15642 TRACE swiftclient http_response_content=body)
2014-11-11 04:35:37.635 15642 TRACE swiftclient ClientException: Container HEAD failed: http://10.108.61.2:8080/v1/AUTH_750fa318e28c4448b32628f860e9d0f9/glance 404 Not Found
2014-11-11 04:35:37.635 15642 TRACE swiftclient
2014-11-11T04:35:37.735910+00:00 debug: 2014-11-11 04:35:37.735 15642 DEBUG swiftclient [3f687a27-501c-4a53-96ba-07c81bc6a8eb f419725e6811497dbcf3550718c0e14c b813afb40d424bec945107c488903d75 - - -] REQ: curl -i http://10.108.61.2:8080/v1/AUTH_750fa318e28c4448b32628f860e9d0f9/glance -X PUT -H "Content-Length: 0" -H "X-Auth-Token: b9e4797e8c784b05bf40dfe36d8eb33c"
 http_log /usr/lib/python2.7/dist-packages/swiftclient/client.py:74
2014-11-11T04:35:37.736993+00:00 debug: 2014-11-11 04:35:37.736 15642 DEBUG swiftclient [3f687a27-501c-4a53-96ba-07c81bc6a8eb f419725e6811497dbcf3550718c0e14c b813afb40d424bec945107c488903d75 - - -] RESP STATUS: 201

Also, there are multiple galeracheck failures http://pastebin.com/EdA5DA9k in logs around as well. As you can see, status 0 if followed by signal 13 (Broken pipe) many times. Perhaps, galera check script should detect such flapping better and mark the mysql backend as down?..
Anyway, looks like glance-api was hit by network flapping and couldn't handle reconnect as appropriate as well

Looks like I'm wrong about network flapping, there are no proof of that in logs. But it is still not clear why galeracheck sometimes report signal 13 and is it normal or not, though, this one should not be related to the glance-api issue anyway

@Bogdan, you're not wrong. It looks like the network setup is the problem here. I've tried to run the same system test locally and also got problems with bond. Current scheme:

        raw_data = {
            'mac': None,
            'mode': 'balance-slb',
            'name': 'ovs-bond0',
            'slaves': [
                {'name': 'eth4'},
                {'name': 'eth3'},
                {'name': 'eth2'},
                {'name': 'eth1'}
            ],
        interfaces = {
            'eth0': ['fuelweb_admin'],
            'ovs-bond0': [
                'public',
                'management',
                'storage',
                'private'
            ]
        }

I.e. NAT forwarded interface eth1 ('public' network) in in the same bond as isolated virtual networks which may cause problems with public network/IPs accessibility.

I suggest to change it to:

        raw_data = {
            'mac': None,
            'mode': 'balance-slb',
            'name': 'ovs-bond0',
            'slaves': [
                {'name': 'eth4'},
                {'name': 'eth3'},
                {'name': 'eth2'},
            ],
            'state': None,
            'type': 'bond',
            'assigned_networks': []
        }

        interfaces = {
            'eth0': ['fuelweb_admin'],
            'eth1': ['public'],
            'ovs-bond0': [
                'management',
                'storage',
                'private'
            ]
        }

I.e. leave NAT forwarded eth0 (admin) and eth1 (public) intact and test bonding on remaning 3 interfaces.

Fix proposed to branch: master
Review: https://review.openstack.org/134890

It still looks like we're using not valid virtual environment for balance-slb testing. We should use virtual switch for this (ovs on the host system side), plug our bond interfaces into that switch and configure it for bonding.

I've faced random network connectivity issues thru the networks attached to the 'ovs-bond0' interface, which is configured in the following way:

* on the slave nodes:
   - eth0 dedicated to 'admin' network;
   - eth1 dedicated to 'public' network;
   - eth2, eth3, eth4 combined into balance-slb bond interface 'ovs-bond0' with tagged networks 'management', 'storage' and 'private';

* on the host:
   - eth2, eth3, eth4 attached to the same libvirt network (the same dobr* bridge on the host)

Doesn't reproduces anymore.
Jenkins jobs that perform bonds testing on CI was configured to use INTERFACE_MODEL=e1000.

ISO version: {u'build_id': u'2015-02-15_22-54-44', u'ostf_sha': u'f9c37d0876141e1550eb4e703a8e500cd463282f', u'build_number': u'126', u'auth_required': True, u'nailgun_sha': u'1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666', u'production': u'docker', u'api': u'1.0', u'python-fuelclient_sha': u'61431ed16fc00039a269424bdbaa410277eff609', u'astute_sha': u'1f87a9b9a47de7498b4061d15a8c7fb9435709d5', u'fuelmain_sha': u'2054229e275d08898b5d079a6625ffcc79ae23b8', u'feature_groups': [u'mirantis'], u'release': u'6.1', u'release_versions': {u'2014.2-6.1': {u'VERSION': {u'build_id': u'2015-02-15_22-54-44', u'ostf_sha': u'f9c37d0876141e1550eb4e703a8e500cd463282f', u'build_number': u'126', u'api': u'1.0', u'nailgun_sha': u'1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666', u'production': u'docker', u'python-fuelclient_sha': u'61431ed16fc00039a269424bdbaa410277eff609', u'astute_sha': u'1f87a9b9a47de7498b4061d15a8c7fb9435709d5', u'feature_groups': [u'mirantis'], u'release': u'6.1', u'fuelmain_sha': u'2054229e275d08898b5d079a6625ffcc79ae23b8', u'fuellib_sha': u'7f8d4382abfcd4338964182ebfea1d539f963e66'}}}, u'fuellib_sha': u'7f8d4382abfcd4338964182ebfea1d539f963e66'}

Change abandoned by Aleksandr Didenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/134890
Reason: No longer needed