Hook error due to cleanup failure if rsync exits early
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| ubuntu-repository-cache (Juju Charms Collection) |
Fix Released
|
Undecided
|
Robert C Jennings | ||
Bug Description
Start hook failure (with sync-on-start=True in config) when initial rsync operations fail. The metadata directory could not be removed during cleanup, causing a traceback (attached).
juju-log:
Performing initial sync from archive.ubuntu.com
Making dir /srv/ubuntu-
...
Archive readiness check for 91.189.92.201
...
/usr/bin/rsync --quiet --links --perms --times --dirs --chmod=g+w --delete --compress --link-
...
rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(251) [Receiver=3.1.0]
rsync failed(12):
...
No retries left. (attempt #5)
Cleaning metadata directory
Removing tree /srv/ubuntu-
Traceback (most recent call last):
File "/var/lib/
HOOKS.
File "/var/lib/
self.
File "/var/lib/
meta_ver = mirror.
File "/var/lib/
result = function(*args, **kwargs)
File "/var/lib/
util.
File "/var/lib/
shutil.
File "/usr/lib/
rmtree(
File "/usr/lib/
onerror(
File "/usr/lib/
os.rmdir(path)
OSError: [Errno 13] Permission denied: '/srv/ubuntu-
ERROR juju.worker.
Related branches
- Marco Ceppi (community): Approve
- Dan Watkins (community): Approve
- Charles Butler: Pending requested
-
Diff: 30 lines (+3/-3)1 file modifiedlib/ubuntu_repository_cache/mirror.py (+3/-3)
| Robert C Jennings (rcj) wrote | #1 |
| Robert C Jennings (rcj) wrote | #2 |
| Changed in ubuntu-repository-cache (Juju Charms Collection): | |
| assignee: | nobody → Robert C Jennings (rcj) |
| status: | New → In Progress |
| description: | updated |
| summary: |
- Sync fails when unable to remove pool directory in metadir + Hook error due to cleanup failure if rsync exits early |
| Changed in ubuntu-repository-cache (Juju Charms Collection): | |
| status: | In Progress → Fix Released |
In the initial failure log we see that the rsync has failed prematurely and repeatedly. As a result, the charm is attempting to clean up the local metadata directory which had been used. When attempting to delete the entire destination metadata tree there is a permission failure which raises an exception. I have not recreated the exact failure as I am not recreating the internal errors that saw rsync exit early, however my tests have created a failure which matches this behavior.
I ran with a locally modified charm that would skip the rsync and fail the mirror_base() function. I that case cleanup failed as we see from the initial environment. Looking at the environment (and code) we see that the destination directory has permissions 0555 when it is create and after a successful rsync they are 0775. I appears that the early exit from rsync leaves the parent directory without write permissions this is causing the clean_metadata_dir function to fail. The fix is to create the directory with user and group write permissions (0o775 rather than 0o555 as we have done).
Summary:
Initial metadata directory permissions blocked clean on failure (LP: #1500442)
If a unit has an rsync problem, the clean_metadata_
with a permission denied error for the metadata directory (or one of its
entries).
Successful 'rsync' execution has a side-effect of changing the metadata
directory permissions from 0555 to 0775. Without rsync success the
directory can not be removed with shutils.rmtree() due to the missing
write permissions on the parent. The initial permissions during directory
creation have been changed to 0775 so that cleanup can proceed even if
rsync has failed.