Openstack Mojo Testing

one-shot-ha specs have hard-coded VIPs (need to consume network env vars)

Bug #1526511 reported by Ryan Beisner
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Openstack Mojo Testing
Fix Released
Undecided
Unassigned

Bug Description

The one-shot specs are hard-coded to 10.5.0.0/16, rather than consuming the VIP and other network env vars that are passed by the test environment. This means that they can only be executed within a bastion instance wired to that subnet, and the spec cannot be automated until this is resolved.

In this example, Glance tries to connect to mysql on an IP which does not exist in the deployment on an address that is not part of the connected networks:
2015-12-15 15:29:41 INFO shared-db-relation-changed 2015-12-15 15:29:41.421 2994 CRITICAL glance [-] OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '10.5.100.1' (113)") None None

Note, that in this test run, the network for the deployed units was 172.17.107.0/24
mysql/0 active idle 1.25.0 18 172.17.107.197 Unit is ready
  mysql-hacluster/0 active idle 1.25.0 172.17.107.197 Unit is ready and clustered

These are the environment variables that were passed to the mojo job. Be aware that there are 20+ different networks, one per jenkins slave, and the mojo spec will need to consume and apply them accordingly.

00:03:40.021 + Setting network env vars:
00:03:40.021 NET_ID: a705dd0f-5571-4818-8c30-4132cc494668
00:03:40.021 GATEWAY: 172.17.107.1
00:03:40.021 NAME_SERVER: 10.5.8.242
00:03:40.021 CIDR_EXT: 172.17.107.0/24
00:03:40.021 FIP_RANGE: 172.17.107.200:172.17.107.249
00:03:40.021 AMULET_OS_VIP: 172.17.107.250
00:03:40.021 AMULET_OS_VIP00: 172.17.107.250
00:03:40.021 AMULET_OS_VIP01: 172.17.107.251
00:03:40.021 AMULET_OS_VIP02: 172.17.107.252
00:03:40.021 AMULET_OS_VIP03: 172.17.107.253

More detail @:
http://paste.ubuntu.com/14030619/

This dynamic variable consumption is already implemented by the non-HA specs which are in automation (full_stack/next, etc). The get_network_env_vars() helper in helper/utils/mojo_utils.py watches for env vars and consumes those to override values from the network.yaml.

See original description
Tags: uosci

Related branches

lp://staging/~thedac/openstack-mojo-specs/ha-automation
Ryan Beisner: Approve
Diff: 676 lines (+121/-56)
16 files modified
helper/bundles/cephradosgw-ha.yaml (+1/-1)
helper/bundles/ha-next.yaml (+9/-9)
helper/bundles/ha.yaml (+9/-9)
helper/bundles/haphase1-next.yaml (+9/-9)
helper/bundles/haphase1.yaml (+9/-9)
helper/bundles/haphase2-next.yaml (+9/-9)
helper/bundles/haphase2.yaml (+9/-9)
helper/bundles/swift-ha-proxy-phase1.yaml (+1/-1)
helper/setup/preflight.py (+44/-0)
specs/full_stack/next_ha_oneshot/icehouse/manifest (+3/-0)
specs/full_stack/next_ha_oneshot/kilo/manifest (+3/-0)
specs/full_stack/next_ha_oneshot/liberty/manifest (+3/-0)
specs/full_stack/next_ha_oneshot/mitaka/manifest (+3/-0)
specs/full_stack/next_ha_oneshot/newton/manifest (+3/-0)
specs/full_stack/next_ha_oneshot/ocata/manifest (+3/-0)
specs/full_stack/next_ha_oneshot/pike/manifest (+3/-0)
Ryan Beisner (1chb1n)
description: updated
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Given that the bundle for the spec uses 9 VIPs, we will need to carve out and pass additional VIP space from test automation.

ha.yaml: vip: 10.5.100.1
ha.yaml: vip: 10.5.100.2
ha.yaml: vip: 10.5.100.3
ha.yaml: vip: 10.5.100.4
ha.yaml: vip: 10.5.100.5
ha.yaml: vip: 10.5.100.9
ha.yaml: vip: 10.5.100.6
ha.yaml: vip: 10.5.100.7
ha.yaml: vip: 10.5.100.8

description: updated
Revision history for this message
Ryan Beisner (1chb1n) wrote :

I've updated UOSCI to pass 14 VIPs instead of the original 4.

For flexibility in consumption, there is now also a new VIP_RANGE env var, formatted as: n.n.n.n:x.x.x.x

Example of new environment variables:
00:01:55.666 + Setting network env vars:
00:01:55.666 NET_ID: a705dd0f-5571-4818-8c30-4132cc494668
00:01:55.666 SUBNET_ID: 1b1f1e9a-0015-4c86-9e29-068e985b7d14
00:01:55.666 GATEWAY: 172.17.107.1
00:01:55.666 NAME_SERVER: 10.5.8.242
00:01:55.666 CIDR_EXT: 172.17.107.0/24
00:01:55.666 FIP_RANGE: 172.17.107.200:172.17.107.239
00:01:55.666 VIP_RANGE: 172.17.107.240:172.17.107.253
00:01:55.666 AMULET_OS_VIP: 172.17.107.240
00:01:55.666 AMULET_OS_VIP00: 172.17.107.240
00:01:55.666 AMULET_OS_VIP01: 172.17.107.241
00:01:55.666 AMULET_OS_VIP02: 172.17.107.242
00:01:55.666 AMULET_OS_VIP03: 172.17.107.243
00:01:55.666 AMULET_OS_VIP04: 172.17.107.244
00:01:55.666 AMULET_OS_VIP05: 172.17.107.245
00:01:55.666 AMULET_OS_VIP06: 172.17.107.246
00:01:55.666 AMULET_OS_VIP07: 172.17.107.247
00:01:55.666 AMULET_OS_VIP08: 172.17.107.248
00:01:55.666 AMULET_OS_VIP09: 172.17.107.249
00:01:55.666 AMULET_OS_VIP10: 172.17.107.250
00:01:55.666 AMULET_OS_VIP11: 172.17.107.251
00:01:55.666 AMULET_OS_VIP12: 172.17.107.252
00:01:55.666 AMULET_OS_VIP13: 172.17.107.253

Revision history for this message
Ryan Beisner (1chb1n) wrote :

Now the spec needs to be adjusted to consume and override VIP values with env vars if they exist.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

Commenting in advance of the solution landing, whilst the thought and paste buffer are fresh :-) ...

For those who run the ha one-shot spec manually, the following, or something like it, must be exported before the mojo spec is run.

It should be noted, however, that even those original IPs are not guaranteed to be safe as they are likely part of the tenant's dhcp pool and will eventually conflict with an instance IP. The safe thing to do is to reduce the dhcp pool size of the tenant's neutron subnet, then export known-free IPs as these VIP values. That is how uosci specifies known-good VIP space outside of the dhcp pool.

export AMULET_OS_VIP00=10.5.100.1
export AMULET_OS_VIP01=10.5.100.2
export AMULET_OS_VIP02=10.5.100.3
export AMULET_OS_VIP03=10.5.100.4
export AMULET_OS_VIP04=10.5.100.5
export AMULET_OS_VIP05=10.5.100.6
export AMULET_OS_VIP06=10.5.100.7
export AMULET_OS_VIP07=10.5.100.8
export AMULET_OS_VIP08=10.5.100.9
export AMULET_OS_VIP09=10.5.100.10
export AMULET_OS_VIP10=10.5.100.11
export AMULET_OS_VIP11=10.5.100.12
export AMULET_OS_VIP12=10.5.100.13
export AMULET_OS_VIP13=10.5.100.14

Revision history for this message
Ryan Beisner (1chb1n) wrote :

All jobs can now use the following equivalent MOJO_* env vars:

MOJO_CIDR_EXT
MOJO_FIP_RANGE
MOJO_GATEWAY
MOJO_NAME_SERVER
MOJO_NET_ID
MOJO_OS_VIP
MOJO_OS_VIP00
MOJO_OS_VIP01
MOJO_OS_VIP02
MOJO_OS_VIP03
MOJO_OS_VIP04
MOJO_OS_VIP05
MOJO_OS_VIP06
MOJO_OS_VIP07
MOJO_OS_VIP08
MOJO_OS_VIP09
MOJO_OS_VIP10
MOJO_OS_VIP11
MOJO_OS_VIP12
MOJO_OS_VIP13
MOJO_SUBNET_ID
MOJO_VIP_RANGE

Revision history for this message
David Ames (thedac) wrote :

For completeness here is the actual MP that landed this fix:
https://code.launchpad.net/~thedac/openstack-mojo-specs/ha-automation/+merge/330464

Chris MacNaughton (chris.macnaughton)
Changed in openstack-mojo-specs:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.