OpenStack Shared File Systems Service (Manila)

NetApp: Temporary failure in name resolution

Bug #1971542 reported by Maurice Escher on 2022-05-04

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Shared File Systems Service (Manila)	Fix Released	Low	Maurice Escher	OpenStack Shared File Systems Service (Manila) zed-rc1

Bug Description

Hi,

sometimes I get failed NetApp API calls due to temporary DNS issues - in my case it is related to the underlying kubernetes coredns, which will also be fixed to have less hickups.

e.g. https://paste.opendev.org/show/beddG0bUVyxLXwPFKGre/

Nevertheless the client can also be improved to handle such temporary issues, since it already makes the impression it is doing multiple retries and gave up only after that.
But looking closer, the default max_retries of the requests underlying urllib3 is actually 0.

I think it makes sense to implement some retries on connect and read calls.
I'll propose a fix.

BR,
Maurice

Tags:

Maurice Escher (maurice-escher) on 2022-05-04

tags:

added: netapp

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-04: Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/840431

Changed in manila:
status:	New → In Progress

Goutham Pacha Ravi (gouthamr) on 2022-07-05

Changed in manila:
importance:	Undecided → Low
assignee:	nobody → Maurice Escher (maurice-escher)
milestone:	none → zed-3

Carlos Eduardo (silvacarlose) on 2022-09-15

Changed in manila:
milestone:	zed-3 → zed-rc1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-09-15: Fix merged to manila (master)

Reviewed: https://review.opendev.org/c/openstack/manila/+/840431
Committed: https://opendev.org/openstack/manila/commit/d3310b0b387727aee17ff1dfe5679c24f8a7c878
Submitter: "Zuul (22348)"
Branch: master

commit d3310b0b387727aee17ff1dfe5679c24f8a7c878
Author: Maurice Escher <email address hidden>
Date: Tue May 3 18:19:25 2022 +0200

[NetApp] Fix lack of retry HTTP requests

    The driver HTTP requester is not handling connection error with some
    retries for avoiding temporary network failure in name resolution.
    This patch adds a custom `HTTPAdapter` with 5 retries, according to
    urrlib3 documentation [1]. Also, the connection retry will be visible
    in the log [2].

[1] https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#module-urllib3.util.retry
[2] https://paste.opendev.org/show/bIcwjk91d4vy5Hnxxs4Z/