The "Too many connections" error can be considered retriable because it's usually temporary. However, when DB drivers raise this error the layers about the driver (sqlalchemy, oslo.db) don't always convert the exception to one of the known exceptions such as sqlalchemy.exc.OperationalError, i.e. the same original exception type can be reused. This is probably a bug in SQLAlchemy but more investigation is needed.
This is a traceback from a real run:
2019-06-15 17:34:52.157 2818 ERROR mistral.engine.post_tx_queue [-] Failed to run transactional engine operation.: OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)
2019-06-15 17:34:52.157 2818 ERROR mistral.engine.post_tx_queue Traceback (most recent call last):
2019-06-15 17:34:52.157 2818 ERROR mistral.engine.post_tx_queue File "/usr/lib/python2.7/site-packages/mistral/engine/post_tx_queue.py", line 119, in _process_tx_queue
2019-06-15 17:34:52.157 2818 ERROR mistral.engine.post_tx_queue func(*args)
2019-06-15 17:34:52.157 2818 ERROR mistral.engine.post_tx_queue File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 285, in _check
2019-06-15 17:34:52.157 2818 ERROR mistral.engine.post_tx_queue wf_handler.check_and_complete(self.wf_ex.id)
2019-06-15 17:34:52.157 2818 ERROR mistral.engine.post_tx_queue File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper
2019-06-15 17:34:52.157 2818 ERROR mistral.engine.post_tx_queue result = f(*args, **kwargs)
2019-06-15 17:34:52.157 2818 ERROR mistral.engine.post_tx_queue File "/usr/lib/python2.7/site-packages/mistral/engine/workflow_handler.py", line 92, in check_and_complete
2019-06-15 17:34:52.157 2818 ERROR mistral.engine.post_tx_queue wf_ex = db_api.load_workflow_execution(wf_ex_id)
....
The corresponding "DB error detected, operation will be retried" message doesn't exist in the log that would be an evidence that an error is retried.
Fix proposed to branch: stable/stein
Review: https:/