Chunksize should be configurable when using copy_from

Bug #1336168 reported by Arnaud Legendre
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Undecided
Arnaud Legendre

Bug Description

When using the copy_from option, readers and writers can have different
speeds to respectively read and write.
A reader timeout will happen if the writer is slow and the writer is
being asked to write a lot. This is currently happening when using
the VMware store and copying from an HTTP server. The reader is reading
16MB which takes too long to upload to vCenter which is causing a
timeout from the HTTP server. The writer should be able to control the
size of the chunks being read when using copy_from: this way the writer
will write fast enough to not make the reader timeout.

This can be reproduced with the filesystem store and copying from http by adding sleep of 30/40 seconds in
https://github.com/openstack/glance/blob/master/glance/store/filesystem.py#L433 between each chunk read.
The chunk will become smaller and then zero.
This is simulating the effect of a slow writer.

Changed in glance:
assignee: nobody → Arnaud Legendre (arnaudleg)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/103753

Changed in glance:
status: New → In Progress
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/103753
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=5148c9648fc959c1d807313176afe3fcf84b89cf
Submitter: Jenkins
Branch: master

commit 5148c9648fc959c1d807313176afe3fcf84b89cf
Author: Arnaud Legendre <email address hidden>
Date: Mon Jun 30 23:47:41 2014 -0700

    Decouple read chunk size from write chunk size

    When using the copy_from option, readers and writers can have different
    speeds to respectively read and write.
    A reader timeout will happen if the writer is slow and the writer is
    being asked to write a lot. This is currently happening when using
    the VMware store and copying from an HTTP server. The reader is reading
    16MB which takes too long to upload to vCenter which is causing a
    timeout from the HTTP server. The writer should be able to control the
    size of the chunks being read when using copy_from: this way the writer
    will write fast enough to not make the reader timeout.

    This patch addresses the issue by introducing the notion of read chunk
    size and write chunk size. Each store can have its own value for read
    and write. The write chunk size of the destination store will be used
    as the read chunk size of the source store in case of an image-create
    where the copy_from option is specified.

    Closes-Bug: #1336168

    Signed-off-by: Arnaud Legendre <email address hidden>
    Signed-off-by: Zhi Yan Liu <email address hidden>

    Change-Id: I4e0c563b8f3a5ced8f65fcca83d341a97729a5d4

Changed in glance:
status: In Progress → Fix Committed
Revision history for this message
Jens Rosenboom (frickler) wrote :

This patch is incomplete and breaks the rbd backend. I get this error with it:

  File "/opt/stack/glance/glance/store/rbd.py", line 153, in __init__
    self.chunk_size = store.chunk_size
AttributeError: 'Store' object has no attribute 'chunk_size'

Revision history for this message
Arnaud Legendre (arnaudleg) wrote :

Thanks for reporting the issue. I have filed another bug (https://bugs.launchpad.net/glance/+bug/1352509) to track this and posted a patch for it https://review.openstack.org/#/c/111798/.

Thierry Carrez (ttx)
Changed in glance:
milestone: none → juno-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in glance:
milestone: juno-3 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.