Looks like a network connectivity issue at node-1 had caused isolation of node-1: 2014-08-14T04:21:05.689938 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:21:05.689938+01:00 notice: (L3_if_downup[eth1](provider=ruby)) Carrier is DOWN, 'eth1' skipping carrier test 2014-08-14T04:21:11.809360 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:21:11.809360+01:00 notice: (L3_if_downup[eth3](provider=ruby)) Carrier is DOWN, 'eth3' skipping carrier test 2014-08-14T04:21:25.132489 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:21:25.132489+01:00 notice: (L3_if_downup[eth4](provider=ruby)) Carrier is DOWN, 'eth4' skipping carrier test 2014-08-14T04:21:31.326668 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:21:31.326668+01:00 notice: (L3_if_downup[eth2](provider=ruby)) Carrier is DOWN, 'eth2' skipping carrier test 2014-08-14T04:21:37.544274 node-1 ./remote/node-1.test.domain.local/puppet-apply.log:2014-08-14T04:21:37.544274+01:00 notice: (L3_if_downup[eth3.103](provider=ruby)) Carrier is DOWN, 'eth3.103' skipping carrier test 2014-08-14T04:29:53.796338 node-1 ./remote/node-1.test.domain.local/haproxy.log:2014-08-14T04:29:53.796338+01:00 alert: Server rabbitmq/node-1 is DOWN, reason: Layer4 connection problem, info: "Network is unreachable", check duration: 0ms. 0 active and 2 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue.
Looks like a network connectivity issue at node-1 had caused isolation of node-1: 14T04:21: 05.689938 node-1 ./remote/ node-1. test.domain. local/puppet- apply.log: 2014-08- 14T04:21: 05.689938+ 01:00 notice: (L3_if_ downup[ eth1](provider= ruby)) Carrier is DOWN, 'eth1' skipping carrier test 14T04:21: 11.809360 node-1 ./remote/ node-1. test.domain. local/puppet- apply.log: 2014-08- 14T04:21: 11.809360+ 01:00 notice: (L3_if_ downup[ eth3](provider= ruby)) Carrier is DOWN, 'eth3' skipping carrier test 14T04:21: 25.132489 node-1 ./remote/ node-1. test.domain. local/puppet- apply.log: 2014-08- 14T04:21: 25.132489+ 01:00 notice: (L3_if_ downup[ eth4](provider= ruby)) Carrier is DOWN, 'eth4' skipping carrier test 14T04:21: 31.326668 node-1 ./remote/ node-1. test.domain. local/puppet- apply.log: 2014-08- 14T04:21: 31.326668+ 01:00 notice: (L3_if_ downup[ eth2](provider= ruby)) Carrier is DOWN, 'eth2' skipping carrier test 14T04:21: 37.544274 node-1 ./remote/ node-1. test.domain. local/puppet- apply.log: 2014-08- 14T04:21: 37.544274+ 01:00 notice: (L3_if_ downup[ eth3.103] (provider= ruby)) Carrier is DOWN, 'eth3.103' skipping carrier test 14T04:29: 53.796338 node-1 ./remote/ node-1. test.domain. local/haproxy. log:2014- 08-14T04: 29:53.796338+ 01:00 alert: Server rabbitmq/node-1 is DOWN, reason: Layer4 connection problem, info: "Network is unreachable", check duration: 0ms. 0 active and 2 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue.
2014-08-
2014-08-
2014-08-
2014-08-
2014-08-
2014-08-