swift_rsyncd upgrade fails due to already mounted disks

Bug #1874691 reported by Rowan Potgieter
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Triaged
High
Unassigned

Bug Description

**Environment**:
* OS : Ubuntu 16.04.6 LTS
* Kernel : 4.4.0-138-generic
* Docker version if applicable : version 19.03.5, build 633a0ea838
* Kolla-Ansible version : stable/stein 4f4c005fa59f3d4810cb9543cd887b5eb5bfb110
* Docker image Install type (source/binary): source and binary, issue is not dependant on type
* Docker image distribution: Ubuntu
* Are you using official images from Docker Hub or self built? Self
* If self built - Kolla version and environment used to build: build.py --base ubuntu --registry my.registry.cloud --push --type binary

**Issue Description**
I recently attempted to upgrade my OpenStack instance from Stein (2 months old) to Stein latest stable and swift failed to upgrade.

The task `Mounting Swift disks` failed because the mount points already existed and the devices were mounted. (I thought ansible mount would handle this?)

In an attempt to fix the issue ran the upgrade again but ignored the mounting error. All the containers were upgraded but the swift_rsynd component did not start. When I investigated further there were I/O errors on /srv/node/d[0-2] which was causing the failure.

I manually unmounted and ran the upgrade targeting swift a 3rd time and everything worked as expected.

To avoid this happening for future upgrades I would suggest unmounting the disks when stopping the container. I have not tested it but I think this would work in roles/swift/tasks/stop.yml

```
- name: Unmounting Swift disks
  become: true
  mount:
    src: "UUID={{ item.fs_uuid }}"
    state: absent
  with_items: "{{ swift_disks }}"
  when: inventory_hostname in groups['swift-account-server'] or
        inventory_hostname in groups['swift-container-server'] or
        inventory_hostname in groups['swift-object-server']
```

I don't believe unmounting would cause any issues since the mounts are only used by the swift_rsyncd container.

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Thanks for the report.

Just to confirm - you used the 'upgrade' kolla-ansible command?
It actually is meant to be used with upgrades between releases, that is between e.g. Rocky and Stein. For updates in one release one should only do 'pull' and 'deploy'.
That said, I am pretty sure the code does not handle well either for swift.

Changed in kolla-ansible:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

self-note TODO: test Swift upgrades

Revision history for this message
Rowan Potgieter (rowan-potgieter) wrote :

Hi Radoslaw

Yes I used the upgrade command.

Sorry I did not realise that upgrade was only for major version releases. I will try the pull and deploy next time.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla-ansible (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/722798

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/722798
Committed: https://opendev.org/openstack/kolla-ansible/commit/b7e85d528292589831c2380ba3f5ed2bd8e98c75
Submitter: "Zuul (22348)"
Branch: master

commit b7e85d528292589831c2380ba3f5ed2bd8e98c75
Author: Radosław Piliszek <email address hidden>
Date: Fri Apr 24 16:14:42 2020 +0200

    [CI] Test Swift upgrades

    Swift is a major OpenStack project. It could use testing upgrades.

    New jobs are placed in the experimental pipeline to avoid
    excessive CI load on general changes.

    Change-Id: I8a089fdd1f21eb4c3e00c38ea9dfcecc77565bf5
    Related-Bug: #1874691

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.