Bug #1351323 “swift-dispersion-populate raises exception” : Bugs : OpenStack Object Storage (swift)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-01: Fix proposed to swift (master)

#1

Fix proposed to branch: master
Review: https://review.openstack.org/111288

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-01: Fix merged to swift (master)

#2

Reviewed: https://review.openstack.org/111288
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8ad37b2217c30b756bb79ead84be7c04f628ed36
Submitter: Jenkins
Branch: master

commit 8ad37b2217c30b756bb79ead84be7c04f628ed36
Author: Christian Schwede <email address hidden>
Date: Fri Aug 1 13:55:21 2014 +0000

Fix swift-dispersion-populate bug when using SimpleClient class

    Commit 7fcbbebb changed the internal client a little bit and removed the
    class variable self.attempts in SimpleClient. This is currently still
    needed in swift-dispersion-populate to track the number of retries, thus
    re-adding it again.

Closes-Bug: 1351323
Change-Id: I98ba6441607158b34708b2dbb2d1a63788681d63

Changed in swift:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-02: Fix proposed to swift (feature/ec)

#3

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/111507

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-02: Change abandoned on swift (feature/ec)

#4

Change abandoned by paul luse (<email address hidden>) on branch: feature/ec
Review: https://review.openstack.org/111507
Reason: accident

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-05: Fix proposed to swift (feature/ec)

#5

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/111866

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-08-05: Fix merged to swift (feature/ec)

#6

Download full text (15.2 KiB)

Reviewed: https://review.openstack.org/111866
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=9ace546b81b7efe606314e24fdb951ffe8e9a253
Submitter: Jenkins
Branch: feature/ec

commit e2255fc1f4d08993f41d3ff2eacc62a3c5f66ecd
Author: David Goetz <email address hidden>
Date: Thu Jul 31 10:19:10 2014 -0700

fix expirer bug with unicode container listings

    If the container names in the expirer's account are returned as
    unicode strings (as is the case with some json libraries), the
    expirer compared eg u'1' == '1', which is problematic. This patch
    ensures that the unicode is coerced to ascii so the comparison
    is correct.

Change-Id: I72b322e7513f7da32e8dc75c6bf0e7e016948c88

commit 4286f36a60446e322f23a329416be3ed67db1149
Author: anc <email address hidden>
Date: Mon Mar 10 11:46:58 2014 +0000

Enable object system metadata on PUTs

    This patch takes a first step towards support
    for object system metadata by enabling headers
    in the x-object-sysmeta- namespace to be
    persisted when objects are PUT. This should be
    useful for other pending patches such as on
    demand migration and server side encryption
    (https://review.openstack.org/#/c/64430/ and
    https://review.openstack.org/#/c/76578/1).

    The x-object-sysmeta- namespace is already
    reserved/protected by the gatekeeper and
    passed through the proxy. This patch modifies
    the object server to persist these headers
    alongside user metadata when an object is
    PUT.

    This patch will preserve existing object
    system metadata and ignore any new system
    metadata when handling object POSTs,
    including POST-as-copy operations. Support
    for modification of object system metadata
    with a POST request requires further work
    as discussed in the blueprint.

    This patch will preserve existing object
    system metadata and update it with new
    system metadata when copying an object.

    A new probe test is added which makes use of
    the BrainSplitter class that has been moved
    from test_container_merge_policy_index.py to
    a new module brain.py.

blueprint object-system-metadata

Change-Id: If716bc15730b7322266ebff4ab8dd31e78e4b962

commit 8ad37b2217c30b756bb79ead84be7c04f628ed36
Author: Christian Schwede <email address hidden>
Date: Fri Aug 1 13:55:21 2014 +0000

Fix swift-dispersion-populate bug when using SimpleClient class

    Commit 7fcbbebb changed the internal client a little bit and removed the
    class variable self.attempts in SimpleClient. This is currently still
    needed in swift-dispersion-populate to track the number of retries, thus
    re-adding it again.

Closes-Bug: 1351323
Change-Id: I98ba6441607158b34708b2dbb2d1a63788681d63

commit 5a84b1cde102dac4ae5a40a20a364ada1dc547e9
Author: Samuel Merritt <email address hidden>
Date: Thu Jul 31 12:18:47 2014 -0700

Put proxy-logging in object expirer's sample config

    It's generally better to have logs for something than to not have
    logs. This way, the object expirer (if using the sample config as a
    starting point) will log wh...

Reviewed:  https://review.openstack.org/111866
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=9ace546b81b7efe606314e24fdb951ffe8e9a253
Submitter: Jenkins
Branch:    feature/ec

commit e2255fc1f4d08993f41d3ff2eacc62a3c5f66ecd
Author: David Goetz <dpgoetz@gmail.com>
Date:   Thu Jul 31 10:19:10 2014 -0700

fix expirer bug with unicode container listings
    
    If the container names in the expirer's account are returned as
    unicode strings (as is the case with some json libraries), the
    expirer compared eg u'1' == '1', which is problematic. This patch
    ensures that the unicode is coerced to ascii so the comparison
    is correct.
    
    Change-Id: I72b322e7513f7da32e8dc75c6bf0e7e016948c88

commit 4286f36a60446e322f23a329416be3ed67db1149
Author: anc <alistair.coles@hp.com>
Date:   Mon Mar 10 11:46:58 2014 +0000

Enable object system metadata on PUTs
    
    This patch takes a first step towards support
    for object system metadata by enabling headers
    in the x-object-sysmeta- namespace to be
    persisted when objects are PUT. This should be
    useful for other pending patches such as on
    demand migration and server side encryption
    (https://review.openstack.org/#/c/64430/ and
    https://review.openstack.org/#/c/76578/1).
    
    The x-object-sysmeta- namespace is already
    reserved/protected by the gatekeeper and
    passed through the proxy. This patch modifies
    the object server to persist these headers
    alongside user metadata when an object is
    PUT.
    
    This patch will preserve existing object
    system metadata and ignore any new system
    metadata when handling object POSTs,
    including POST-as-copy operations. Support
    for modification of object system metadata
    with a POST request requires further work
    as discussed in the blueprint.
    
    This patch will preserve existing object
    system metadata and update it with new
    system metadata when copying an object.
    
    A new probe test is added which makes use of
    the BrainSplitter class that has been moved
    from test_container_merge_policy_index.py to
    a new module brain.py.
    
    blueprint object-system-metadata
    
    Change-Id: If716bc15730b7322266ebff4ab8dd31e78e4b962

commit 8ad37b2217c30b756bb79ead84be7c04f628ed36
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Fri Aug 1 13:55:21 2014 +0000

Fix swift-dispersion-populate bug when using SimpleClient class
    
    Commit 7fcbbebb changed the internal client a little bit and removed the
    class variable self.attempts in SimpleClient. This is currently still
    needed in swift-dispersion-populate to track the number of retries, thus
    re-adding it again.
    
    Closes-Bug: 1351323
    Change-Id: I98ba6441607158b34708b2dbb2d1a63788681d63

commit 5a84b1cde102dac4ae5a40a20a364ada1dc547e9
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Jul 31 12:18:47 2014 -0700

Put proxy-logging in object expirer's sample config
    
    It's generally better to have logs for something than to not have
    logs. This way, the object expirer (if using the sample config as a
    starting point) will log what it does.
    
    Note that the container reconciler's sample config already contains
    proxy-logging, as does the proxy server's. The object expirer is the
    odd man out.
    
    Change-Id: I32aac99131746501820319b94405440c1934a694

commit 4bc600ff42f254849458c77ddef9bdc47f91e4a1
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Jul 25 18:50:19 2014 -0700

Fix "Handoff requested (N)" log spam
    
    A long, long time ago, on a GET request, the proxy would go look on 3*
    nodes for the requested thing. If one of the primary nodes was
    error-limited, it'd look on two primaries and a handoff. Since this
    indicated some failure somewhere, the proxy would emit a warning:
    "Handoff requested (1)". If two primaries were down, there'd be a
    second message "Handoff requested (2)", and so on.
    
    Some StatsD messages were emitted too.
    
    A somewhat shorter time ago (commit d79a67eb), the proxy started
    looking into handoffs if it got 404s from the primaries. While this
    was a good idea, it resulted lots of "Handoff requested (N)" log spam;
    you'd see these messages on every single 404. Also, the StatsD
    handoff_count and handoff_all_count metrics shot way up and turned
    into noise.
    
    This commit restores the original intent (and usefulness) of the log
    messages and StatsD metrics: if the proxy only looks at the normal
    number of handoff nodes, nothing is logged. However, if a primary is
    down, then the message "Handoff requested (1)" will be logged,
    indicating that the proxy looked at one more handoff than it normally
    would, and this happened because a primary node was error-limited.
    
    Closes-Bug: 1297214
    
    * or whatever the replica count was
    
    Change-Id: If1b77c18c880b096e8ab1df3008db40ce313835d

commit 1cd14daadb40f5d76e559d874e106e9b63612689
Author: paul luse <paul.e.luse@intel.com>
Date:   Fri Jul 25 16:34:39 2014 -0700

Reuse a cursor instead of a conn.execute for better performance
    
    Basically the same thing as https://review.openstack.org/#/c/109694/
    but over in the account backend.
    
    Change-Id: I9255f3145818da3adafc16d3bcf159608a78fe51

commit 670dc920deaf64a99f6ca3ad10ba19029493f3ff
Author: David Goetz <dpgoetz@gmail.com>
Date:   Fri Jul 25 12:16:54 2014 -0700

Reuse a cursor instead of a conn.execute for better performance
    
    In some limited testing it was ~10% faster. Every little bit helps right?
    
    Change-Id: Ide55b7600d625ad151daca73fbdd8684f0f9e7f8

commit 6aff48c6f1effc445633764e9d78b6f7c17f8b7e
Author: anc <alistair.coles@hp.com>
Date:   Mon Jul 7 12:32:44 2014 +0100

Fix trivial typos
    
    Fixes a few typos I have stumbled across recently.
    
    Change-Id: Ib232924f6b23c08578c52a8dd63aaaa8789f9da7

commit 665c95331f9d0a8b16a28ea3efafc7ba20da414c
Author: anc <alistair.coles@hp.com>
Date:   Thu Jul 17 11:17:57 2014 +0100

Update doc for list_endpoints v2 API
    
    Adds some v2 doc to list_endpoints module.
    
    Also adds LE to list of swift sources in logs.rst
    
    Change-Id: I6ccbcf677b12f5aa7db0f9d81e43ce5ca990cc57

commit ee14d58d84e6dfb2b6bf007a18a78a49d080db6e
Author: guang-yee <guang.yee@hp.com>
Date:   Wed Jul 23 10:27:40 2014 -0700

Use auth_token from keystonemiddleware
    
    auth_token middleware in python-keystoneclient is deprecated and has
    been moved to the keystonemiddleware repo.
    
    Change-Id: Ia04aa83348e0776cb3239cb5420ee1450a990d5b
    Closes-Bug: #1342274

commit f918bd47c04a6b46b92f97a0e34cc205bee5bd00
Author: Lin Yang <lin.a.yang@intel.com>
Date:   Wed Jul 23 17:31:06 2014 +0800

Remove duplicate conditional statement
    
    It contains two same conditional statements to check whether the first
    argurment is end with '.ring.gz'. The latter one can be covered by 'else'
    clause.
    
    Change-Id: I4b67dc330e82e793020bec4b74047062313ae11c
    Signed-off-by: Lin Yang <lin.a.yang@intel.com>

commit 05121f533a22c8ebe7bd893ba8d0014e3e69e929
Author: ppalucki <pawel.palucki@gmail.com>
Date:   Tue Jul 22 12:30:54 2014 +0200

typo tradeoff -> trade-off
    
    Change-Id: Iad5eb611bc727991606f8a2d4dbb08dcc8fa3ddb

commit 02aaabba6a1ff91abbf07ee4ed0804dafa873341
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Jul 21 11:37:57 2014 -0700

Fix a couple small things in formpost doc string
    
    Change-Id: Idb8d4707180396662493aacbd1f125c4ca43a1ca

commit b908a65649cbdd0ed5d09d11fb635c36b5f35bc1
Author: Michael Barton <mike@weirdlooking.com>
Date:   Mon Jul 21 07:45:56 2014 +0000

lock_file race fixes
    
    I attempted to use this function and found a few problems.
    
    We shouldn’t unlink the file after closing it, because someone else could lock
    it in between.  Switch to unlink before close.
    
    If someone else locked the file between our open and flock, they are likely to
    unlink it out from underneath us.  Then we have a lock on a file that no longer
    exists.  So stat the filename after locking to make sure the inode hasn't
    changed or gone away.
    
    We probably shouldn’t unlink the file if we time out waiting for a lock.  So
    move that to before the finally block.
    
    Change-Id: Id1858c97805d3ab81c584eaee8ce0d43d34a8089

commit 90272a0564e3824e1199fb7e1250de124c1f34b0
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Wed Jul 2 17:21:54 2014 +1000

Add X-Delete-At/After support to FormPost
    
    This change adds the ability to specify a X-Delete-At or
    X-Delete-After attribute when using the FormPost middleware.
    
    Here is an example of what you need to add to the form:
    
      <input type="hidden" name="x_delete_at" value="<unix-timestamp>"/>
      <input type="hidden" name="x_delete_after" value="<seconds>"/>
    
    To be inline with the other form imput names, x-delete-at/after has
    changed to x_delete_at/after.
    
    DocImpact
    Change-Id: Ib1cc0bcf1dd7d2b689f2f26d100f9bab36880c81
    Closes-Bug: #1065522

commit 116ac459a64471b09a5e989e88349c66f470ce38
Author: paul luse <paul.e.luse@intel.com>
Date:   Fri Jul 18 14:35:12 2014 -0700

Fix small typo in deployment guide
    
    Change-Id: Ifd86ce5b36c2588b57fd44c633b095d1f49fcd5b

commit e1fb64a7a6c675d909fe76757ae6671e51de9a51
Author: David Moreau Simard <dmsimard@iweb.com>
Date:   Fri Jul 18 13:25:58 2014 -0400

Pass disk usage options when using --all
    
    This commit allows the arguments --human-readable and --top to be
    used with --all to customize the output of the disk usage portion.
    
    There are no unit tests attached to this commit, I have created and
    referenced bug #1344200 which shows there is a larger bug to address
    to create several missing tests.
    
    Change-Id: I46b8359533989efc3067971acaafec1d0cbc2b9f
    Closes-Bug: #1344178
    Related-Bug: #1344200

commit 216aaab6387cd4db63aeb51520e8c1b9c70e31a5
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Jun 20 15:43:40 2014 -0700

Add v2 API to list endpoints middleware
    
    The new API adds better support for storage policies and changes the
    response from a list of backend urls to a dictionary with a key
    "endpoints" that's a list of of the backend urls and a new key headers
    that's a dictionary of headers to send along with the backend request.
    In the v2 response format for object requests, there the headers key
    includes "X-Backend-Storage-Policy-Index" which indicates the storage
    policy index for the endpoints returned in the response.
    
    Change-Id: I706a5b5be8002c633fe97b2429256735800a902e

commit 25c98b771a26ffed3772e2fe4d2547afaa011834
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Jul 11 12:03:30 2014 -0700

Allow HEAD for POST tempurls
    
    HEAD requests are already allowed for GET and PUT tempurls; this
    commit adds that for POST tempurls. Since POST replaces all the
    object's metadata, it's quite useful to be able to HEAD the object in
    order to fetch the old metadata and do a client-side merge of the new
    metadata (like with normal, token-authenticated POST requests).
    
    Change-Id: I603c7822cd27f0e304fd27024f83f95114eb0aef

commit e5d90a9cc3f784d23a4248a50a99287af6b99cc3
Author: Thiago da Silva <thiago@redhat.com>
Date:   Fri Jul 11 11:13:52 2014 -0400

clean up saio env. after runnning func tests
    
    Added call to delete container in teardown_package().
    Tests are leaving objects and containers in saio environment
    after finishing running the tests.
    
    Currently, delete_containers() is called in the setUp of each test
    class, which cleans up the containers of the previous test,
    but leaves the containers of the last test in the saio environment.
    
    Change-Id: I643d7083a2a310fc3d24eab48c565f3798cff25f
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit e5d76e9f055805e1b44e4ee2f1898a5b9a0a354c
Author: Gil Vernik <gilv@il.ibm.com>
Date:   Thu Apr 24 09:50:36 2014 +0300

Container PUT requests and max container per account
    
    If container counter per account is equal or greater than
    max_container_per_account, then all PUT requests are failed and
    403 is returned.
    This is correct behaviour if the request is to create a new
    container, however if container already exists PUT should be
    allowed, even the max_container_per_account condition has met.
    
    This patch allows to process PUT requests for existing containers,
    even if max_container_per_account > = container count.
    
    It indirectly resolve the bug 1306711, since swift-client
    uses internally PUT requests for container, prior it upload an
    object there.
    
    Change-Id: I2dcf20b6feb27e346111466a565695eba4b4b1da

commit 7573fbd4985fb090cc315b5af9e7531dc900b7b8
Author: Steven Lang <Steven.Lang@hgst.com>
Date:   Mon Jun 23 12:59:24 2014 -0700

Object services user-agent string uses full name
    
    It does not appear that, aside from the user-agent string, the strings
    "obj-server", "obj-updater", or "obj-replicator" (or "obj-<anything>"*)
    appear in the swift code base, aside from the directory containing the
    object services code being named "obj".
    
    Furthermore, the container, account, and proxy services construct their
    user-agent string, as reported in the logs, using their full name. In
    addition, this full name also shows up as the name of the process via
    "ps" or "top", etc., which can make it easier for admins to match log
    entries with other tools.
    
    For consistency, we update the object services to use an "object-"
    prefix rather than "obj-" in its user agent string.
    
    * obj-etag does appear in a unit test, but not part of the regular
    code.
    
    Change-Id: I914fc189514207df2535731eda10cb4b3d30cc6c

commit 3cad20570c79ec4b817b6998dc2e63bed1ea8c1d
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Jun 25 20:34:39 2014 -0700

Add X-Backend-Timestamp on more object server responses
    
    It's particularly interesting on writes (PUT, POST, DELETE) where the
    current on-disk timestamp would prevent the object server from serving
    the incoming request and returns 409 Conflict.
    
    The FakeConn has also been updated to respond in kind for 409's on
    expect and all responses generaly just cause it's good to keep fakes in
    line with the reals - not that I expected any existing tests to break
    because of the new headers.
    
    Change-Id: Iac6fbd2f872a9521bb2db84a333365b69f54fb6c

commit f2774a4d11bb005c0d54928477ad7b82afbf0cf3
Author: MORITA Kazutaka <morita.kazutaka@gmail.com>
Date:   Tue Apr 22 09:28:15 2014 +0900

Disable case-changing behavior in Eventlet
    
    RFC 2616 says that HTTP header fields are case-insensitive.  However, there are
    some S3 clients who don't accept normalized header by Swift and Eventlet.  For
    example, AWS Java SDK expects that an etag header is 'ETag', not 'Etag'.
    
    This patch disables Eventlet's header capitalization so that the swift3
    middleware can normalize the response headers as those clients expect.
    
    Note that this change requires a fix for Eventlet, which will be included in
    the next Eventlet release (v0.15).
    
    Change-Id: I6d3428b0dafef776bdb3ebac7639b3126fa5e60d

Thierry Carrez (ttx) on 2014-08-25

Changed in swift:
milestone:	none → 2.1.0
status:	Fix Committed → Fix Released

OpenStack Object Storage (swift)

swift-dispersion-populate raises exception

Bug Description

Other bug subscribers

Remote bug watches