Compass

neutron api gets unreachable behind ha proxy intermittently

Bug #1344144 reported by Jerry Zhao on 2014-07-18

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Compass	New	Undecided	Compass Team

Bug Description

172.16.0.1:60373 [18/Jul/2014:04:00:19.536] neutron_api neutron_api/host1.1 0/0/0/144/145 201 469 - - ---- 1/0/0/0/0 0/0 "POST /v2.0/networks.json HTTP/1.1"
172.16.0.1:60374 [18/Jul/2014:04:00:19.683] neutron_api neutron_api/host1.1 0/0/0/118/119 201 341 - - ---- 1/0/0/0/0 0/0 "POST /v2.0/routers.json HTTP/1.1"
Server neutron_api/host1.1 is DOWN, reason: Layer7 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
Server neutron_api/host1.1 is DOWN, reason: Layer7 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
proxy neutron_api has no server available!
proxy neutron_api has no server available!

2014-07-18 04:00:26.800 | -------------------- >> begin captured logging << --------------------
2014-07-18 04:00:26.801 | neutronclient.client: DEBUG:
2014-07-18 04:00:26.801 | REQ: curl -i http://172.16.0.253:9696/v2.0/networks.json -X POST -H "X-Auth-Token: MIIJPAYJKoZIhvcNAQcCoIIJLTCCCSkCAQExCTAHBgUrDgMCGjCCCBUGCSqGSIb3DQEHAaCCCAYEgggCeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wNy0xOFQxMTowMDoxNy45ODc1MTQiLCAiZXhwaXJlcyI6ICIyMDE0LTA3LTE5VDExOjAwOjE3WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAiLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo4Nzc2L3YxLzAxMTVkODQ3YTgyNDQxNTA4M2RlNzFkYWViOGI3ZTAwIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo4Nzc2L3YxLzAxMTVkODQ3YTgyNDQxNTA4M2RlNzFkYWViOGI3ZTAwIiwgImlkIjogIjE1NTNmNzgyODU0MzRhOTliMmQwYTMzYTg2NDM4Mzg2IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjg3NzYvdjEvMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo5MjkyL3YyIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo5MjkyL3YyIiwgImlkIjogImY1NDgwNjc3ZTNmMTQ5Y2ZhODE5MGNiMDFmMjM5M2M1IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjkyOTIvdjIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjg3NzQvdjIvMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjg3NzQvdjIvMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAiLCAiaWQiOiAiYTViNTBiNzBjOWExNGQ1NWFhYTA4MDRhMGY5NTEyNGYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6ODc3NC92Mi8wMTE1ZDg0N2E4MjQ0MTUwODNkZTcxZGFlYjhiN2UwMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6OTY5NiIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6OTY5NiIsICJpZCI6ICJjNjlhZWJmZWUzOTg0NmNiODY0ZDA4MWQ5YzZlYzI0OSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJxdWFudHVtIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1MzozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo1MDAwL3YyLjAiLCAiaWQiOiAiYjkwZWM3ZGRiNGQ3NDU0Yzk0ODY4MmM1NGU3OTViOGYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJjb25zb2xlIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICIzYzMzNjFlYTU3OTc0OWZmODRkYTU2M2JlYzM1NDYwZiIsICJyb2xlcyI6IFt7Im5hbWUiOiAiYWRtaW4ifV0sICJuYW1lIjogImNvbnNvbGUifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiMWQ4ZTUxYjZkNWU3NDFmNWJhMzcyMDVjMjQzNGU4ODEiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCr8jcF+uPOChBb51W0kpoLbhOTxNkohDdScpGIIBx3flTILnYo2gLIVDrMqKJZoTMsY1+s2MpDsCYh2ruFqqQTBD7c-A6v9+uBYbB4PZ6SYYvZd8lu+EgyxKN1KM0T6FyAWYPS6vHKgYqzWkZRajHP2Zcd14XoFYpGGYEYofCV4A==" -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-neutronclient" -d '{"network": {"tenant_id": "7831f62269b540e5813956cd604de606", "name": "network-smoke-848329"}}'
2014-07-18 04:00:26.801 |
2014-07-18 04:00:26.801 | neutronclient.client: DEBUG: RESP:{'date': 'Fri, 18 Jul 2014 11:00:19 GMT', 'status': '201', 'content-length': '340', 'content-type': 'application/json; charset=UTF-8'} {"network": {"status": "ACTIVE", "subnets": [], "name": "network-smoke-848329", "provider:physical_network": null, "admin_state_up": true, "tenant_id": "7831f62269b540e5813956cd604de606", "provider:network_type": "gre", "router:external": false, "shared": false, "id": "c861f847-4717-4d42-ab3e-035ad41a5004", "provider:segmentation_id": 3}}
2014-07-18 04:00:26.801 |
2014-07-18 04:00:26.801 | tempest.test: DEBUG: Adding {u'status': u'ACTIVE', u'subnets': [], u'name': u'network-smoke-848329', u'provider:physical_network': None, u'admin_state_up': True, u'tenant_id': u'7831f62269b540e5813956cd604de606', u'provider:network_type': u'gre', u'router:external': False, u'shared': False, u'id': u'c861f847-4717-4d42-ab3e-035ad41a5004', u'provider:segmentation_id': 3} to shared resources of TestNetworkBasicOps
2014-07-18 04:00:26.801 | neutronclient.client: DEBUG:
2014-07-18 04:00:26.802 | REQ: curl -i http://172.16.0.253:9696/v2.0/routers.json -X POST -H "X-Auth-Token: MIIJPAYJKoZIhvcNAQcCoIIJLTCCCSkCAQExCTAHBgUrDgMCGjCCCBUGCSqGSIb3DQEHAaCCCAYEgggCeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wNy0xOFQxMTowMDoxNy45ODc1MTQiLCAiZXhwaXJlcyI6ICIyMDE0LTA3LTE5VDExOjAwOjE3WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAiLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo4Nzc2L3YxLzAxMTVkODQ3YTgyNDQxNTA4M2RlNzFkYWViOGI3ZTAwIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo4Nzc2L3YxLzAxMTVkODQ3YTgyNDQxNTA4M2RlNzFkYWViOGI3ZTAwIiwgImlkIjogIjE1NTNmNzgyODU0MzRhOTliMmQwYTMzYTg2NDM4Mzg2IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjg3NzYvdjEvMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo5MjkyL3YyIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo5MjkyL3YyIiwgImlkIjogImY1NDgwNjc3ZTNmMTQ5Y2ZhODE5MGNiMDFmMjM5M2M1IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjkyOTIvdjIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjg3NzQvdjIvMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjg3NzQvdjIvMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAiLCAiaWQiOiAiYTViNTBiNzBjOWExNGQ1NWFhYTA4MDRhMGY5NTEyNGYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6ODc3NC92Mi8wMTE1ZDg0N2E4MjQ0MTUwODNkZTcxZGFlYjhiN2UwMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6OTY5NiIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6OTY5NiIsICJpZCI6ICJjNjlhZWJmZWUzOTg0NmNiODY0ZDA4MWQ5YzZlYzI0OSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJxdWFudHVtIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1MzozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo1MDAwL3YyLjAiLCAiaWQiOiAiYjkwZWM3ZGRiNGQ3NDU0Yzk0ODY4MmM1NGU3OTViOGYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJjb25zb2xlIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICIzYzMzNjFlYTU3OTc0OWZmODRkYTU2M2JlYzM1NDYwZiIsICJyb2xlcyI6IFt7Im5hbWUiOiAiYWRtaW4ifV0sICJuYW1lIjogImNvbnNvbGUifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiMWQ4ZTUxYjZkNWU3NDFmNWJhMzcyMDVjMjQzNGU4ODEiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCr8jcF+uPOChBb51W0kpoLbhOTxNkohDdScpGIIBx3flTILnYo2gLIVDrMqKJZoTMsY1+s2MpDsCYh2ruFqqQTBD7c-A6v9+uBYbB4PZ6SYYvZd8lu+EgyxKN1KM0T6FyAWYPS6vHKgYqzWkZRajHP2Zcd14XoFYpGGYEYofCV4A==" -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-neutronclient" -d '{"router": {"tenant_id": "7831f62269b540e5813956cd604de606", "name": "router-smoke-84057", "admin_state_up": true}}'
2014-07-18 04:00:26.802 |
2014-07-18 04:00:26.802 | neutronclient.client: DEBUG: RESP:{'date': 'Fri, 18 Jul 2014 11:00:19 GMT', 'status': '201', 'content-length': '212', 'content-type': 'application/json; charset=UTF-8'} {"router": {"status": "ACTIVE", "external_gateway_info": null, "name": "router-smoke-84057", "admin_state_up": true, "tenant_id": "7831f62269b540e5813956cd604de606", "id": "08420181-4ab9-4a31-872b-059894fcfbcc"}}
2014-07-18 04:00:26.802 |
2014-07-18 04:00:26.802 | tempest.test: DEBUG: Adding {u'status': u'ACTIVE', u'external_gateway_info': None, u'name': u'router-smoke-84057', u'admin_state_up': True, u'tenant_id': u'7831f62269b540e5813956cd604de606', u'id': u'08420181-4ab9-4a31-872b-059894fcfbcc'} to shared resources of TestNetworkBasicOps
2014-07-18 04:00:26.802 | neutronclient.client: DEBUG:
2014-07-18 04:00:26.802 | REQ: curl -i http://172.16.0.253:9696/v2.0/routers/08420181-4ab9-4a31-872b-059894fcfbcc.json -X PUT -H "X-Auth-Token: MIIJPAYJKoZIhvcNAQcCoIIJLTCCCSkCAQExCTAHBgUrDgMCGjCCCBUGCSqGSIb3DQEHAaCCCAYEgggCeyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxNC0wNy0xOFQxMTowMDoxNy45ODc1MTQiLCAiZXhwaXJlcyI6ICIyMDE0LTA3LTE5VDExOjAwOjE3WiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAiLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo4Nzc2L3YxLzAxMTVkODQ3YTgyNDQxNTA4M2RlNzFkYWViOGI3ZTAwIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo4Nzc2L3YxLzAxMTVkODQ3YTgyNDQxNTA4M2RlNzFkYWViOGI3ZTAwIiwgImlkIjogIjE1NTNmNzgyODU0MzRhOTliMmQwYTMzYTg2NDM4Mzg2IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjg3NzYvdjEvMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo5MjkyL3YyIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo5MjkyL3YyIiwgImlkIjogImY1NDgwNjc3ZTNmMTQ5Y2ZhODE5MGNiMDFmMjM5M2M1IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjkyOTIvdjIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjg3NzQvdjIvMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTcyLjE2LjAuMjUzOjg3NzQvdjIvMDExNWQ4NDdhODI0NDE1MDgzZGU3MWRhZWI4YjdlMDAiLCAiaWQiOiAiYTViNTBiNzBjOWExNGQ1NWFhYTA4MDRhMGY5NTEyNGYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6ODc3NC92Mi8wMTE1ZDg0N2E4MjQ0MTUwODNkZTcxZGFlYjhiN2UwMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6OTY5NiIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6OTY5NiIsICJpZCI6ICJjNjlhZWJmZWUzOTg0NmNiODY0ZDA4MWQ5YzZlYzI0OSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJxdWFudHVtIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1MzozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE3Mi4xNi4wLjI1Mzo1MDAwL3YyLjAiLCAiaWQiOiAiYjkwZWM3ZGRiNGQ3NDU0Yzk0ODY4MmM1NGU3OTViOGYiLCAicHVibGljVVJMIjogImh0dHA6Ly8xNzIuMTYuMC4yNTM6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJjb25zb2xlIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICIzYzMzNjFlYTU3OTc0OWZmODRkYTU2M2JlYzM1NDYwZiIsICJyb2xlcyI6IFt7Im5hbWUiOiAiYWRtaW4ifV0sICJuYW1lIjogImNvbnNvbGUifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiMWQ4ZTUxYjZkNWU3NDFmNWJhMzcyMDVjMjQzNGU4ODEiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYCr8jcF+uPOChBb51W0kpoLbhOTxNkohDdScpGIIBx3flTILnYo2gLIVDrMqKJZoTMsY1+s2MpDsCYh2ruFqqQTBD7c-A6v9+uBYbB4PZ6SYYvZd8lu+EgyxKN1KM0T6FyAWYPS6vHKgYqzWkZRajHP2Zcd14XoFYpGGYEYofCV4A==" -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-neutronclient" -d '{"router": {"external_gateway_info": {"network_id": "726be239-7dc1-4b97-9140-f4cf8437d5d0"}}}'
2014-07-18 04:00:26.803 |
2014-07-18 04:00:26.803 | neutronclient.client: DEBUG: RESP:{'status': '502', 'connection': 'close', 'content-type': 'text/html', 'cache-control': 'no-cache'} <html><body><h1>502 Bad Gateway</h1>
2014-07-18 04:00:26.803 | The server returned an invalid or incomplete response.
2014-07-18 04:00:26.803 | </body></html>
2014-07-18 04:00:26.803 |
2014-07-18 04:00:26.803 |

Revision history for this message

Jerry Zhao (zhaoxinyu) wrote on 2014-07-18:

http://12.234.32.44/logs/compass-bare-daily-7vm/2014-07-18_02-51-00/cobbler_logs/host1.1/quantum-server.log
or
http://12.234.32.44/logs/compass-bare-daily-7vm/2014-07-18_02-51-00/cobbler_logs/host1.1/quantum-server.log.gz

Changed in compass:
assignee:	nobody → Compass Team (syscompass)

Revision history for this message

Jerry Zhao (zhaoxinyu) wrote on 2014-07-18:

Download full text (3.7 KiB)

Probably still caused by rpc timeout to schedule l3 agent task.

2014-07-18 04:01:21 ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.6/site-packages/quantum/common/rpc.py", line 43, in dispatch
    quantum_ctxt, version, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.6/site-packages/quantum/db/agents_db.py", line 173, in report_state
    plugin.create_or_update_agent(context, agent_state)
  File "/usr/lib/python2.6/site-packages/quantum/db/agents_db.py", line 156, in create_or_update_agent
    context.session.add(agent_db)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 402, in __exit__
    self.commit()
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 314, in commit
    self._prepare_impl()
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 298, in _prepare_impl
    self.session.flush()
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1583, in flush
    self._flush(objects)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1654, in _flush
    flush_context.execute()
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 331, in execute
    rec.execute(self)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 475, in execute
    uow
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 59, in save_obj
    mapper, table, update)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 485, in _emit_update_statements
    execute(statement, params)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
  ...

Probably still caused by rpc timeout to schedule l3 agent task.

2014-07-18 04:01:21    ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
 Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.6/site-packages/quantum/common/rpc.py", line 43, in dispatch
    quantum_ctxt, version, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.6/site-packages/quantum/db/agents_db.py", line 173, in report_state
    plugin.create_or_update_agent(context, agent_state)
  File "/usr/lib/python2.6/site-packages/quantum/db/agents_db.py", line 156, in create_or_update_agent
    context.session.add(agent_db)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 402, in __exit__
    self.commit()
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 314, in commit
    self._prepare_impl()
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 298, in _prepare_impl
    self.session.flush()
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1583, in flush
    self._flush(objects)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1654, in _flush
    flush_context.execute()
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 331, in execute
    rec.execute(self)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 475, in execute
    uow
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 59, in save_obj
    mapper, table, update)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 485, in _emit_update_statements
    execute(statement, params)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
 OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'UPDATE agents SET heartbeat_timestamp=%s WHERE agents.id = %s' (datetime.datetime(2014, 7, 18, 11, 0, 20, 5080), '3d198f04-8054-4c5b-8864-b4ea51f6f172')
 2014-07-18 04:01:21    DEBUG [quantum.policy] loading policy file at /etc/quantum/policy.json

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-18: Related fix proposed to compass-adapters (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/108111

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-21: Related fix merged to compass-adapters (master)

Reviewed: https://review.openstack.org/108111
Committed: https://git.openstack.org/cgit/stackforge/compass-adapters/commit/?id=7283866a8a139fec9a4a52c463162b9395570e90
Submitter: Jenkins
Branch: master

commit 7283866a8a139fec9a4a52c463162b9395570e90
Author: Jerry Zhao <email address hidden>
Date: Fri Jul 18 10:30:25 2014 -0700

Increase timeout connect to allow for longer wait time.

    When backend gets extremely busy especially such complex system as
    openstack, increase timeout to allow for longer tcp connnection time
    before the backend is deemed dead.
    Refer to: http://docs.openstack.org/high-availability-guide/content/
    ha-aa-haproxy.html

Change-Id: I03f5937424dc77a94eec0c03a890668a15358e01
Related-bug: #1344144

Revision history for this message

Jerry Zhao (zhaoxinyu) wrote on 2014-07-23:

http://12.234.32.44/logs/compass-bare-daily-7vm/2014-07-23_02-51-00/cobbler_logs/host4.1/haproxy.log

172.16.0.50:28342 [23/Jul/2014:04:00:56.835] keystone_admin keystone_admin/host1.1 0/0/0/18/30 200 283 - - ---- 2/0/0/0/0 0/0 "GET /v2.0/tenants/a8031f6f355241499f9909e3b21e8c58/users HTTP/1.1"
Server neutron_api/host1.1 is DOWN, reason: Layer7 timeout, check duration: 2000ms. 0 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
Server neutron_api/host1.1 is DOWN, reason: Layer7 timeout, check duration: 2000ms. 0 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
proxy neutron_api has no server available!
proxy neutron_api has no server available!

http://12.234.32.44/logs/compass-bare-daily-7vm/2014-07-23_02-51-00/cobbler_logs/host1.1/quantum-server.log
There didn't show any sign of dead lock this time in quantum api, so it might have been a false negative. Just that quantum was busy. Try allowing more wait time for httpchk and using the recommended value in openstack doc.
http://docs.openstack.org/high-availability-guide/content/ha-aa-haproxy.html

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-07-23:

Reviewed: https://review.openstack.org/109049
Committed: https://git.openstack.org/cgit/stackforge/compass-adapters/commit/?id=d4b335deddb7c9fea59d4a0a93520e8ecfedc1b0
Submitter: Jenkins
Branch: master

commit d4b335deddb7c9fea59d4a0a93520e8ecfedc1b0
Author: Jerry Zhao <email address hidden>
Date: Wed Jul 23 10:20:53 2014 -0700

Add timeout check and timeout queue with recommended value

    There were also occurances that httpchk experienced timeout. Right
    now we are using default value 2000ms. Try using 10s suggested by
    the example config in openstack doc.

Related-bug: #1344144
Change-Id: I7a72d7a22a1845d1e2d480eee88246f980f87e6c

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.