cinder-backup uses surprisingly high memory

Any advices on how to find out what is actually using those memory? Current solution is still rebooting cinder-backup service.

Hi norman,

Can you tell which backend is used for backup and which backend for volumes?

The cause for the high RSS even when idle is a high water mark memory issue caused by the per native thread glibc malloc arena.

I'm proposing a devstack patch to fix it https://review.opendev.org/c/openstack/devstack/+/845805 and a tripleo one as well https://review.opendev.org/c/openstack/tripleo-common/+/845807

You can try it in your deployment yourself running the cinder-backup service with those environmental variables.

thank you very much for the input. I will try it asap. Besides, may I have some references explaining its meanings? And are those tunables available for all glibc versions? what would be the potential tradeoff for those tunables? thank you again and looking forward to the reply.

Hi Rajat,

sorry for the delay. we are using swift backend to talk to a ceph rgw cluster. we do have customized
the code.

best,
Norman

Description of the tunable parameters can be seen on the glibc manual: https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html

I don't know the exact versions where each configuration parameter was added and fully supported, but I see that the M_ARENA_MAX environmental variable was added back in 2009 (https://sourceware.org/git/?p=glibc.git;a=commit;h=425ce2edb9d11cc1ff650fac16dfbc450241896a), so probably in glibc version 2.10

In my opinion there should be no trade-off for Python programs, since the independent arenas are meant to reduce congestion (due to the locking) for multi-threaded applications that do a considerable amount of memory reserve and free operations, but in Python we have the GIL, so we are already suffering the penalties of the locking and won't benefit from that performance improvement.