swift_rsyncd upgrade fails due to already mounted disks
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
kolla-ansible |
Triaged
|
High
|
Unassigned |
Bug Description
**Environment**:
* OS : Ubuntu 16.04.6 LTS
* Kernel : 4.4.0-138-generic
* Docker version if applicable : version 19.03.5, build 633a0ea838
* Kolla-Ansible version : stable/stein 4f4c005fa59f3d4
* Docker image Install type (source/binary): source and binary, issue is not dependant on type
* Docker image distribution: Ubuntu
* Are you using official images from Docker Hub or self built? Self
* If self built - Kolla version and environment used to build: build.py --base ubuntu --registry my.registry.cloud --push --type binary
**Issue Description**
I recently attempted to upgrade my OpenStack instance from Stein (2 months old) to Stein latest stable and swift failed to upgrade.
The task `Mounting Swift disks` failed because the mount points already existed and the devices were mounted. (I thought ansible mount would handle this?)
In an attempt to fix the issue ran the upgrade again but ignored the mounting error. All the containers were upgraded but the swift_rsynd component did not start. When I investigated further there were I/O errors on /srv/node/d[0-2] which was causing the failure.
I manually unmounted and ran the upgrade targeting swift a 3rd time and everything worked as expected.
To avoid this happening for future upgrades I would suggest unmounting the disks when stopping the container. I have not tested it but I think this would work in roles/swift/
```
- name: Unmounting Swift disks
become: true
mount:
src: "UUID={{ item.fs_uuid }}"
state: absent
with_items: "{{ swift_disks }}"
when: inventory_hostname in groups[
```
I don't believe unmounting would cause any issues since the mounts are only used by the swift_rsyncd container.
Thanks for the report.
Just to confirm - you used the 'upgrade' kolla-ansible command?
It actually is meant to be used with upgrades between releases, that is between e.g. Rocky and Stein. For updates in one release one should only do 'pull' and 'deploy'.
That said, I am pretty sure the code does not handle well either for swift.