list os-service causing reconnects to memcached all the time

Bug #1995029 reported by norman shen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Triaged
Wishlist
Unassigned

Bug Description

Description
===========

we are running a victoria openstack cluster (python3). and I observe that everytime when an openstack compute service list executed, nova-api will create a new connection to memcache. Actually there are several reasons to cause this behavior

1. when running natively with eventlet's wsgi server, everytime a new coroutine is created to host web request and this causes keystonemiddle auth_token which uses python-memcached to reconnect to memcahced all the time
2. os-services will trigger nova.availability_zones.set_availability_zones and it will update cache every time, since cellv2 is enabled, this method is running in an co-routine as well
3. python-memcached's Client is inheriting from threading.local which will be monkey_patched to use eventlet's implementation and thus for every co-routine context it will create a new connection

Steps to reproduce
==================

1. Patch def _get_socket and print connection
2. execute openstack compute service list

Expected result
===============

Maintain stable connections to memcached

Actual result
=============

Reconnects

Environment
===========

1. devstack victoria openstack

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

This is a valid performance improvement. Feel free to push a patch to implement it.

Changed in nova:
status: New → Triaged
importance: Undecided → Wishlist
tags: added: performance
tags: added: low-hanging-fruit
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.