Fuel for OpenStack

  1. Bug #1394195
  2. Comment #7

Comment 7 for bug 1394195

Revision history for this message
Artem Panchenko (apanchenko-8) wrote :

This issue also affects 5.1.x, I reproduced it during performance testing: under high load keystone on one controller crashed and begun to return 504 error. But HAProxy still continued to send requests to it:

root@node-2:~# echo "show stat" | nc -U /var/lib/haproxy/stats | awk -F',' '{if (NR == 1 || $1 ~/keystone/){printf "%14s %14s %10s %10s %10s %10s %10s %10s %10s %10s %10s\n", $1,$2,$8,$13,$14,$15,$18,$22,$23,$25,$28}}'
      # pxname svname stot ereq econ eresp status chkfail chkdown downtime iid
    keystone-1 FRONTEND 12898 0 OPEN 3
    keystone-1 node-2 4308 0 769 UP 0 0 0 3
    keystone-1 node-3 4295 0 0 UP 0 1 809 3
    keystone-1 node-6 4295 0 0 UP 0 1 817 3
    keystone-1 BACKEND 12898 0 769 UP 0 0 3
    keystone-2 FRONTEND 135668 0 OPEN 4
    keystone-2 node-2 44786 0 8305 UP 147 16 191 4
    keystone-2 node-3 46007 0 6 UP 0 1 807 4
    keystone-2 node-6 46011 0 8 UP 0 1 817 4
    keystone-2 BACKEND 135668 0 8319 UP 0 0 4

So lots of attempts to create new instances failed:

2014-12-02 17:52:14.069 20474 INFO rally.benchmark.runners.base [-] Task b40d77e3-e267-4fa2-9bdd-9f10ccb9c68e | ITER: 846 END: Error GetResourceFailure: Failed to get the resource <Server: rally_novaserver_jpxlzicqprsefvsy>: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-6829db25-b851-44db-aa6b-2f37dbb890c3)
2014-12-02 17:52:14.069 20474 INFO rally.benchmark.runners.base [-] Task b40d77e3-e267-4fa2-9bdd-9f10ccb9c68e | ITER: 923 START
2014-12-02 17:52:14.247 20476 INFO rally.benchmark.runners.base [-] Task b40d77e3-e267-4fa2-9bdd-9f10ccb9c68e | ITER: 818 END: Error GatewayTimeout: Gateway Timeout (HTTP 504)