retry seems doesn't work in memcached driver when exceptions occur

Bug #1975439 reported by zhen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tooz
New
Undecided
Unassigned

Bug Description

The script is:
```python
from tooz import coordination
from tooz import _retry
from tooz.drivers import memcached as mem1
from oslo_utils import timeutils
import uuid
import time
import tenacity
from tenacity import *
import logging
import threading
import functools
import pdb

LOG = logging.getLogger(__name__)

pdb.set_trace()

LOCK_TIMEOUT = 30
url = "memcached://100.101.89.222:11211"
member_id = ("cinder-" + str(uuid.uuid4())).encode('ascii')
coordinator = coordination.get_coordinator(
    url, member_id, lock_timeout=LOCK_TIMEOUT
)
coordinator.start(start_heart=True)

lock_name = "uuid-delete_snasphot".encode("ascii")

# to get a lock
lock = coordinator.get_lock(lock_name)

@_retry.retry(stop_max_delay=True)
@mem1._translate_failures
def _acquire():
    print("trying add lock %s", lock.name)
    if lock.coord.client.add(
            lock.name,
            lock.coord._member_id,
            expire=LOCK_TIMEOUT,
            noreply=False):
        print("add lock success")
        return True
    raise _retry.TryAgain

# to acquire lock
_acquire()

lock.coord.client.delete(lock.name)
lock.release()
coordinator.stop()

```

when lock was got and was about to acquire lock, i shutdown the memcache server '100.101.89.222:11211', It never tried acquire again.

the result is:
```bash
(Pdb)
> /home/lz/tes-memcache.py(24)<module>()
-> coordinator = coordination.get_coordinator(
(Pdb)
> /home/lz/tes-memcache.py(25)<module>()
-> url, member_id, lock_timeout=LOCK_TIMEOUT
(Pdb)
> /home/lz/tes-memcache.py(27)<module>()
-> coordinator.start(start_heart=True)
(Pdb)
> /home/lz/tes-memcache.py(29)<module>()
-> lock_name = "uuid-delete_snasphot".encode("ascii")
(Pdb) c
trying add lock %s b'__TOOZ_LOCK_uuid-delete_snasphot'
Traceback (most recent call last):
  File "/home/lz/my_project_env/lib/python3.6/site-packages/tooz/drivers/memcached.py", line 43, in _failure_translator
    yield
  File "/home/lz/my_project_env/lib/python3.6/site-packages/tooz/drivers/memcached.py", line 70, in wrapper
    return func(*args, **kwargs)
  File "tes-memcache.py", line 47, in _acquire
    noreply=False):
  File "/home/lz/my_project_env/lib/python3.6/site-packages/pymemcache/client/base.py", line 1169, in add
    flags=flags)
  File "/home/lz/my_project_env/lib/python3.6/site-packages/pymemcache/client/base.py", line 393, in add
    flags=flags)[key]
  File "/home/lz/my_project_env/lib/python3.6/site-packages/pymemcache/client/base.py", line 942, in _store_cmd
    buf, line = _readline(self.sock, buf)
  File "/home/lz/my_project_env/lib/python3.6/site-packages/pymemcache/client/base.py", line 1266, in _readline
    raise MemcacheUnexpectedCloseError()
pymemcache.exceptions.MemcacheUnexpectedCloseError
```

When a temporary exception(like memcachedunexpectedERROR or brokenpipe) occur, is retry strategy "retry_if_exception_type" more proper than "retry_never" here?

zhen (zhen001)
summary: - retry seems doesn't work in memcached driver when trying to acquire lock
+ retry seems doesn't work in memcached driver when exceptions occur
description: updated
Revision history for this message
zhen (zhen001) wrote (last edit ):

# tooz._retry.retry

```python
import tenacity
from tenacity import stop
from tenacity import wait

_default_wait = wait.wait_exponential(max=1)

def retry(stop_max_delay=None, **kwargs):
    k = {"wait": _default_wait, "retry": tenacity.retry_never}
    if stop_max_delay not in (True, False, None):
        k['stop'] = stop.stop_after_delay(stop_max_delay)
    return tenacity.retry(**k)

TryAgain = tenacity.TryAgain
```

Does ``"retry": tenacity.retry_never`` necessary?

description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.