MagnetoDB

API Service errors during the load testing: status code 500 and 502

Bug #1273214 reported by Timur Nurlygayanov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MagnetoDB
Won't Fix
Medium
Unassigned
MagnetoDB 2014.2 "juno"

Bug Description

Magneto DB API service returns incorrect responses during the performance testing.

When we perform 800+ request per second to API service, we can see several responses with status code '500' and status code '502'.
As a result about 6 % of responses have statuses with error codes, when we perform 1000 request per second.

(please, see attached screenshot with performance tests results)

Error messages in magnetodb log files after the testing (also added to the bug description in comments):
http://paste.openstack.org/show/61928/
http://paste.openstack.org/show/61925/

We have send the following requests to MagnetoDB API service:
195
POST / HTTP/1.0
User-Agent: yandex-tank/1.1.1
Host: 172.18.169.205:8080
Content-Length:2
X-Amz-Target: DynamoDB_20120810.ListTables
Content-Type: application/x-amz-json-1.0
Connection: Close

{}

316
POST / HTTP/1.0
User-Agent: yandex-tank/1.1.1
Host: 172.18.169.205:8080
Content-Length:123
X-Amz-Target: DynamoDB_20120810.GetItem
Content-Type: application/x-amz-json-1.0
Connection: Close

{"TableName": "load_spam_table", "Key": {"user_id": {"S": "<email address hidden>"}, "date_message_id": {"S": "2013-12-31#123456"}}}

See original description
Tags: performance
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :
Download full text (10.1 KiB)

Errors in Magneto DB Logs:

2014-01-21 06:13:32.622 8448 TRACE gunicorn.error
2014-01-21 06:13:32.630 8448 ERROR gunicorn.error [-] Error handling request
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error Traceback (most recent call last):
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/gunicorn/workers/async.py", line 45, in handle
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error self.handle_request(listener, req, client, addr)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/gunicorn/workers/ggevent.py", line 151, in handle_request
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error super(GeventWorker, self).handle_request(*args)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/gunicorn/workers/async.py", line 93, in handle_request
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error respiter = self.wsgi(environ, resp.start_response)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error resp = self.call_func(req, *args, **self.kwargs)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error return self.func(req, *args, **kwargs)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/magnetodb/common/wsgi.py", line 50, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error response = req.get_response(self.application)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error application, catch_exc_info=False)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error app_iter = application(self.environ, start_response)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error return resp(environ, start_response)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error response = self.app(environ, start_response)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error return resp(environ, start_response)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error resp = self.call_func(req, *args, **self.kwargs)
2014-01-21 06:13:32.630 8448 TRACE gunicorn.error File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
2014-01-21 06:13:32.630 8...

description: updated
Ilya Sviridov (isviridov)
information type: Public → Private
information type: Private → Public Security
information type: Public Security → Public
Ilya Sviridov (isviridov)
Changed in magnetodb:
milestone: none → next
importance: Undecided → Medium
Timur Nurlygayanov (tnurlygayanov)
Changed in magnetodb:
status: New → Confirmed
Ilya Sviridov (isviridov)
Changed in magnetodb:
status: Confirmed → Invalid
milestone: next → juno-rc1
status: Invalid → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.