OpenStack Shared File Systems Service (Manila)

login share server maybe fail with private key during creating a share

Bug #1748139 reported by wang yong
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Wishlist
Pavlo Shchelokovskyy

Bug Description

When using the GenericShareDriver and creating a share, the create routine maybe failed after the new share server created, and during to login it with private key to initialize the share server, especially on a slow test environment node.

The code only check the server availability with connect to the port, however, even when the port is reachable, the login maybe failed, we should check the login too.

share backend config in /etc/manila/manila.conf:

[generic1]
admin_subnet_id = d904feb9-470a-45a1-97c8-d6adbf653211
admin_network_id = 41c38256-dcfa-4ed8-b8f6-dba210ef91be
driver_handles_share_servers = True
service_instance_user = manila
service_image_name = manila-service-image-master
path_to_private_key = /opt/stack/.ssh/id_rsa
path_to_public_key = /opt/stack/.ssh/id_rsa.pub
share_backend_name = GENERIC1
share_driver = manila.share.drivers.generic.GenericShareDriver

wang yong (wangyong2017)
Changed in manila:
assignee: nobody → wang yong (wangyong2017)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/542739

OpenStack Infra (hudson-openstack)
Changed in manila:
assignee: wang yong (wangyong2017) → Tom Barron (tpb)
Tom Barron (tpb)
Changed in manila:
assignee: Tom Barron (tpb) → nobody
Revision history for this message
Victoria Martinez de la Cruz (vkmc) wrote :

Checked during Manila bug squash, seems that the proposed fix is valid, would need to be rebased and reviewers comments addressed. Bug needs to be reproduced with the condition that reporter indicated (slow environment). If anybody is interested on picking this one up, you are encouraged to do that!

Changed in manila:
importance: Undecided → Wishlist
Goutham Pacha Ravi (gouthamr)
Changed in manila:
status: In Progress → Triaged
Pavlo Shchelokovskyy (pshchelo)
Changed in manila:
assignee: nobody → Pavlo Shchelokovskyy (pshchelo)
OpenStack Infra (hudson-openstack)
Changed in manila:
status: Triaged → In Progress
Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

I've proposed a new patch https://review.opendev.org/c/openstack/manila/+/939068
this utilizes already existing SSHPool to instantiate SSH connection.

As a general remark, we do seem to catch this quite often on our downstream CI. The symptoms are manila-tempest-plugin tests failing with "Share XXX failed to build and is in ERROR status" and the like,
and the following error in the manila logs from paramiko - "encountered RSA key, expected OPENSSH key" (which is a general error when login is not working and paramiko just retries login with parsing the key as various possible key types returning the error for the last key type checked https://github.com/paramiko/paramiko/issues/1655#issuecomment-607628569)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on manila (master)

Change abandoned by "Goutham Pacha Ravi <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/manila/+/542739
Reason: being revived here: https://review.opendev.org/c/openstack/manila/+/939068

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.opendev.org/c/openstack/manila/+/939068
Committed: https://opendev.org/openstack/manila/commit/688ebca877ce17ba549900cecee06b6a053f7883
Submitter: "Zuul (22348)"
Branch: master

commit 688ebca877ce17ba549900cecee06b6a053f7883
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Fri Jan 10 12:19:52 2025 +0000

    Actually try to log in via SSH to validate server

    socket on SSH port being open may not be enough, as the cloud init
    might've not yet setup the user keys by this point.

    Instead, actually try to connect via SSH with password/key
    to validate everything is up and ready to accept commands.

    Change-Id: Icd12ba54cadfd0b85a1768054a4f7b2537ed07ac
    Closes-Bug: #1748139

Changed in manila:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.