Object Node timeouts retrieving EC segments returns 404 to client
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Confirmed
|
Medium
|
Unassigned |
Bug Description
When pushing an Object Server hard Getting EC objects timeouts result in the proxy sending a 404 to the client. Most clients do not retry on a 404, but would retry on a 503. Previous to 2.5, I was receiving a number of 503s during the same testing. These have been changed to 404s, which many applications take as the object does not exist or was deleted.
Here are the logs from my proxy server during a period with Object Server disk busy do to benchmark testing.
Oct 28 18:15:58 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 18:15:58 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 18:15:58 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 18:15:58 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 18:15:58 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 18:15:58 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 18:15:58 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 18:15:58 localhost proxy-server: 192.168.2.41 192.168.2.41 28/Oct/
X-Auth-Token: ECTesting,
User-Agent: python-
Connection: keep-alive
X-Trans-Id: txf5b7019b37a74
X-Backend-
Host: 192.168.2.51
Accept: */*
Accept-Encoding: gzip, deflate
Content-Type: None 1.0855 - - 1446070557.
Here are my logs before Swift 2.5 (note I was still getting 404s, but I was also getting 503s in the same testing)
Oct 28 19:58:43 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 19:58:43 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 19:58:43 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 19:58:43 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 19:58:43 localhost proxy-server: ERROR with Object server 192.168.
Oct 28 19:58:47 localhost proxy-server: 192.168.2.41 192.168.2.41 28/Oct/
Oct 28 19:58:52 localhost proxy-server: 192.168.2.41 192.168.2.41 28/Oct/
Doug
Changed in swift: | |
status: | New → Confirmed |
importance: | Undecided → Low |
Changed in swift: | |
importance: | Low → Medium |
So, I shutdown half of my saio object servers and was able to reproduce this:
Nov 3 22:15:15 saio proxy-server: ERROR with Object server 127.0.0.1:6040/sdb8 re: Trying to GET /v1/AUTH_ test/ec- test/test: Connection refused 2015:22: 15:15 +0000] "GET /sdb2/941/ AUTH_test/ ec-test/ test" 200 82 "GET http:// saio:8080/ v1/AUTH_ test/ec- test/test" "tx6be44b4f68c5 4c0f881f9- 00563931f3" "proxy-server 27232" 0.0024 "-" 27264 1 test/ec- test/test: Connection refused test/ec- test/test: Connection refused 2015:22: 15:15 +0000] "GET /sdb1/941/ AUTH_test/ ec-test/ test" 200 82 "GET http:// saio:8080/ v1/AUTH_ test/ec- test/test" "tx6be44b4f68c5 4c0f881f9- 00563931f3" "proxy-server 27232" 0.0008 "-" 27205 1 2015:22: 15:15 +0000] "GET /sdb5/941/ AUTH_test/ ec-test/ test" 200 82 "GET http:// saio:8080/ v1/AUTH_ test/ec- test/test" "tx6be44b4f68c5 4c0f881f9- 00563931f3" "proxy-server 27232" 0.0011 "-" 27205 1 2015:22: 15:15 +0000] "GET /sdb6/941/ AUTH_test/ ec-test/ test" 404 - "GET http:// saio:8080/ v1/AUTH_ test/ec- test/test" "tx6be44b4f68c5 4c0f881f9- 00563931f3" "proxy-server 27232" 0.0004 "-" 27264 1 test/ec- test/test: Connection refused 2015/22/ 15/15 GET /v1/AUTH_ test/ec- test/test HTTP/1.0 404 - python- swiftclient- 2.6.1.dev10 AUTH_tk84450147e... - 70 - tx6be44b4f68c54 c0f881f9- 00563931f3 - 0.0284 - - 1446588915. 392549038 1446588915. 420908928 1
Nov 3 22:15:15 saio object-6020: 127.0.0.1 - - [03/Nov/
Nov 3 22:15:15 saio proxy-server: ERROR with Object server 127.0.0.1:6040/sdb4 re: Trying to GET /v1/AUTH_
Nov 3 22:15:15 saio proxy-server: ERROR with Object server 127.0.0.1:6030/sdb7 re: Trying to GET /v1/AUTH_
Nov 3 22:15:15 saio object-6010: 127.0.0.1 - - [03/Nov/
Nov 3 22:15:15 saio object-6010: 127.0.0.1 - - [03/Nov/
Nov 3 22:15:15 saio object-6020: 127.0.0.1 - - [03/Nov/
Nov 3 22:15:15 saio proxy-server: ERROR with Object server 127.0.0.1:6030/sdb3 re: Trying to GET /v1/AUTH_
Nov 3 22:15:15 saio proxy-server: 127.0.0.1 127.0.0.1 03/Nov/
Oddly if I disable 3/4 object servers - it get the 503's:
Nov 3 22:17:21 saio object-6010: 127.0.0.1 - - [03/Nov/ 2015:22: 17:21 +0000] "GET /sdb5/941/ AUTH_test/ ec-test/ test" 200 82 "GET http:// saio:8080/ v1/AUTH_ test/ec- test/test" "txc2605e264169 4ae69f1d2- 0056393271" "proxy-server 27232" 0.0009 "-" 27205 1 2015:22: 17:21 +0000] "GET /sdb1/941/ AUTH_test/ ec-test/ test" 200 82 "GET http:// saio:8080/ v1/AUTH_ test/ec- test/test" "txc2605e264169 4ae69f1d2- 0056393271" "proxy-server 27232" 0.0006 "-" 27205 1 test/ec- test/test: Connection refused test/ec- test/test: Connection refused test/ec- test/test: Connection refused test/ec- test/test: Connection refused
Nov 3 22:17:21 saio object-6010: 127.0.0.1 - - [03/Nov/
Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6040/sdb4 re: Trying to GET /v1/AUTH_
Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6040/sdb8 re: Trying to GET /v1/AUTH_
Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6030/sdb7 re: Trying to GET /v1/AUTH_
Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6020/sdb2 re: Trying to GET /v1/AUTH_
Nov 3 22:17:21 saio proxy-server: ERROR with Object server 127.0.0.1:6...