Ceph/NFS driver hits ganesha export ID limit
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Shared File Systems Service (Manila) |
In Progress
|
Low
|
Ashley Rodriguez |
Bug Description
Description
===========
It is not possible to create new access rules for the Manila shares backed by CephFS/NFS when the driver hits the export ID limit:
We create a new access rule:
```
$ manila access-allow testnfs1 ip 0.0.0.0
+------
| Property | Value |
+------
| id | 84d1c08d-
| share_id | 38ffd41c-
| access_level | rw |
| access_to | 0.0.0.0 |
| access_type | ip |
| state | queued_to_apply |
| access_key | None |
| created_at | 2023-02-
| updated_at | None |
| metadata | {} |
+------
```
But its in error state and not possible to access to the new nfs filesystem:
```
# manila access-list testnfs1
+------
| id | access_type | access_to | access_level | state | access_key | created_at | updated_at |
+------
| 84d1c08d-
+------
```
Error from Ganesha logs:
```
...
...
27/02/2023 13:24:07 : epoch 63fc7e79 : controller11 : ganesha.
27/02/2023 13:24:07 : epoch 63fc7e79 : controller11 : ganesha.
27/02/2023 13:24:07 : epoch 63fc7e79 : controller11 : ganesha.
Config File (/etc/ganesha/
Config File (/etc/ganesha/
Config File (/etc/ganesha/
```
Specifically:
Config File (/etc/ganesha/
RCA:
"Exporting" (creating a share and allowing access to it) more than 64536 NFS shares with CephFS/NFS can cause this issue. A single ganesha gateway would possibly be overwhelmed (and overwhelm the hardware it is running on) with those many shares. We don't have to keep the shares around, we could export and delete them over a period of time. The bug here is that the driver never reclaims old export IDs.
Steps to reproduce
==================
An easy way to reproduce the failure is to set the export index counter rados object to a very high number
*
echo "65536" | sudo tee -a /etc/ganesha/
sudo rados -n client.manila -p manila_data put ganesha-
sudo rm /etc/ganesha/
* Create a new share and allow-access to it
Expected result
===============
Access rules are successfully applied
Actual result
=============
Access rules transition to "error" state and the share is not exported via NFS-Ganesha.
Environment
===========
1. Version of OpenStack Manila: Train, the issue persists with master however
2. Storage backend: CephFS (NFS)
tags: | added: cephfs |
tags: | added: ganesha |
Changed in manila: | |
importance: | Undecided → Low |
milestone: | none → bobcat-1 |
Changed in manila: | |
status: | New → Triaged |
Changed in manila: | |
milestone: | bobcat-1 → bobcat-2 |
Changed in manila: | |
assignee: | nobody → Ashley Rodriguez (ashrod98) |
Changed in manila: | |
milestone: | bobcat-2 → bobcat-3 |
Changed in manila: | |
milestone: | bobcat-3 → caracal-1 |
This issue has a workaround ..
Additional related discussions are at https:/ /meetings. opendev. org/meetings/ manila/ 2023/manila. 2023-03- 16-15.00. log.html