Glance cache cleaner wrongly deletes cache for non invalid images

Bug #1228256 reported by Sulochan Acharya
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
High
Feilong Wang

Bug Description

The cache cleaner is currently cleaning invalid images like so:

def clean(self, stall_time=None):
        """
        Delete any image files in the invalid directory and any
        files in the incomplete directory that are older than a
        configurable amount of time.
        """
        self.delete_invalid_files()

        if stall_time is None:
            stall_time = CONF.image_cache_stall_time

        now = time.time()
        older_than = now - stall_time
        self.delete_stalled_files(older_than)

def delete_stalled_files(self, older_than):
        """
        Removes any incomplete cache entries older than a
        supplied modified time.

        :param older_than: Files written to on or before this timestemp
                           will be deleted.
        """
        for path in self.get_cache_files(self.incomplete_dir):
            os.unlink(path)
            LOG.info(_("Removed stalled cache file %s"), path)

Although the "older_than" time is passed its not used anywhere, as a result if you run this through a cron, valid images in process for download/getting cached will be deleted from /incomplete since the cleaner does not check for time at all.

Glance api log will complain when this happens in the middle of the fetch:

Fetch finished, moving '/var/lib/glance/image-cache/incomplete/b9b38c90-cf64-4fa9-9e73-73da77e40074' to '/var/lib/glance/image-cache/b9b38c90-cf64-4fa9-9e73-73da77e40074' commit

ERROR glance.image_cache [...] Exception encountered while tee'ing image 'b9b38c90-cf64-4fa9-9e73-73da77e40074' into cache: [Errno 2] No such file or directory. Continuing with response.

Feilong Wang (flwang)
Changed in glance:
assignee: nobody → Fei Long Wang (flwang)
Feilong Wang (flwang)
Changed in glance:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/47866

Changed in glance:
importance: Undecided → Medium
importance: Medium → High
milestone: none → havana-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/47866
Committed: http://github.com/openstack/glance/commit/29d4383b166a77595df1b2a24b02a5b2556aaba2
Submitter: Jenkins
Branch: master

commit 29d4383b166a77595df1b2a24b02a5b2556aaba2
Author: Fei Long Wang <email address hidden>
Date: Mon Sep 23 11:45:57 2013 +0800

    Cache cleaner wrongly deletes cache for non invalid images

    Based on current implement of method delete_stalled_files,
    it's just deleting all the images located in /incomplete
    instead of using the param 'older_than' to check if it
    should be deleted. So this fix will use 'older_than' to
    confirm brfore deleting.

    Fixes bug 1228256

    Change-Id: Ica27915f8b9b098ecae1e38c8e678d6acf4f2f76

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in glance:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in glance:
milestone: havana-rc1 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.