In attached logs I found that nova had got 504 from haproxy. haproxy logs for cinder have a lot of 504 answers
ogubanov@ogubanov-pc:~/Downloads/logs$ find ./docker-logs/ -name haproxy.log | xargs grep 'cinder' | grep -c ' 504 ' 161
and I found that haproxy (seems) reset POST requests if it takes >= 60 second
ogubanov@ogubanov-pc:~/Downloads/logs$ find ./docker-logs/ -name haproxy.log | xargs grep 'cinder' | grep ' 504 ' | tail -n 3 ./docker-logs/node-20.domain.tld/haproxy.log:2016-01-12T09:50:37.451451+00:00 info: 192.168.0.35:49739 [12/Jan/2016:09:49:37.453] cinder-api cinder-api/node-20 0/0/0/-1/60001 504 194 - - sH-- 1848/24/24/24/0 0/0 "POST /v1/fcf23cb570d640ffb4c93d4dcca9c138/volumes/20d52a22-efdd-4848-b82e-c9a347510adb/action HTTP/1.1" ./docker-logs/node-20.domain.tld/haproxy.log:2016-01-12T09:51:06.319330+00:00 info: 192.168.0.23:47543 [12/Jan/2016:09:50:06.300] cinder-api cinder-api/node-20 0/0/0/-1/60011 504 194 - - sH-- 1862/23/23/23/0 0/0 "POST /v1/fcf23cb570d640ffb4c93d4dcca9c138/volumes/4f453d0f-6602-476c-8330-1ef1caa91aa1/action HTTP/1.1" ./docker-logs/node-20.domain.tld/haproxy.log:2016-01-12T09:51:06.330370+00:00 info: 192.168.0.36:41157 [12/Jan/2016:09:50:06.302] cinder-api cinder-api/node-20 0/0/0/-1/60021 504 194 - - sH-- 1861/22/22/22/0 0/0 "POST /v1/fcf23cb570d640ffb4c93d4dcca9c138/volumes/f19c61b7-ebc8-40cc-af57-1840a59ae428/action HTTP/1.1"
Details here http://pastebin.com/MjWu9sMr
In attached logs I found that nova had got 504 from haproxy.
haproxy logs for cinder have a lot of 504 answers
ogubanov@ ogubanov- pc:~/Downloads/ logs$ find ./docker-logs/ -name haproxy.log | xargs grep 'cinder' | grep -c ' 504 '
161
and I found that haproxy (seems) reset POST requests if it takes >= 60 second
ogubanov@ ogubanov- pc:~/Downloads/ logs$ find ./docker-logs/ -name haproxy.log | xargs grep 'cinder' | grep ' 504 ' | tail -n 3 logs/node- 20.domain. tld/haproxy. log:2016- 01-12T09: 50:37.451451+ 00:00 info: 192.168.0.35:49739 [12/Jan/ 2016:09: 49:37.453] cinder-api cinder-api/node-20 0/0/0/-1/60001 504 194 - - sH-- 1848/24/24/24/0 0/0 "POST /v1/fcf23cb570d 640ffb4c93d4dcc a9c138/ volumes/ 20d52a22- efdd-4848- b82e-c9a347510a db/action HTTP/1.1" logs/node- 20.domain. tld/haproxy. log:2016- 01-12T09: 51:06.319330+ 00:00 info: 192.168.0.23:47543 [12/Jan/ 2016:09: 50:06.300] cinder-api cinder-api/node-20 0/0/0/-1/60011 504 194 - - sH-- 1862/23/23/23/0 0/0 "POST /v1/fcf23cb570d 640ffb4c93d4dcc a9c138/ volumes/ 4f453d0f- 6602-476c- 8330-1ef1caa91a a1/action HTTP/1.1" logs/node- 20.domain. tld/haproxy. log:2016- 01-12T09: 51:06.330370+ 00:00 info: 192.168.0.36:41157 [12/Jan/ 2016:09: 50:06.302] cinder-api cinder-api/node-20 0/0/0/-1/60021 504 194 - - sH-- 1861/22/22/22/0 0/0 "POST /v1/fcf23cb570d 640ffb4c93d4dcc a9c138/ volumes/ f19c61b7- ebc8-40cc- af57-1840a59ae4 28/action HTTP/1.1"
./docker-
./docker-
./docker-
Details here http:// pastebin. com/MjWu9sMr