Ubuntu
swift package

metadata update should not delete large file manifest

Bug #1388880 reported by Chris Stratford
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
python-swiftclient
New
Undecided
Unassigned
swift (Ubuntu)
Triaged
Low
Unassigned

Bug Description

If you upload a large file into swift using "swift -S" (or presumably the upload() method) then add extra metadata using "swift post -m" (or post_object()), the large file manifest metadata is deleted. It shouldn't do that.

$ swift upload -S 10000000 chrisstest /boot/initrd.img-3.16.0-22-generic
boot/initrd.img-3.16.0-22-generic segment 3
boot/initrd.img-3.16.0-22-generic segment 0
boot/initrd.img-3.16.0-22-generic segment 1
boot/initrd.img-3.16.0-22-generic segment 2
/boot/initrd.img-3.16.0-22-generic

$ swift stat chrisstest boot/initrd.img-3.16.0-22-generic
       Account: AUTH_06b6c15ea8c54b8f9832cbaf1ccccfb6
     Container: chrisstest
        Object: boot/initrd.img-3.16.0-22-generic
  Content Type: application/octet-stream
Content Length: 30730085
 Last Modified: Mon, 03 Nov 2014 14:51:20 GMT
          ETag: "235bbbe8ab0520eec94a6bb8cc079db6"
      Manifest: chrisstest_segments/boot/initrd.img-3.16.0-22-generic/1413445437.000000/30730085/10000000/
    Meta Mtime: 1413445437.000000
 Accept-Ranges: bytes
   X-Timestamp: 1415026308.79456

$ swift post -m "x-object-meta-test1:test1" chrisstest boot/initrd.img-3.16.0-22-generic

$ swift stat chrisstest boot/initrd.img-3.16.0-22-generic
                 Account: AUTH_06b6c15ea8c54b8f9832cbaf1ccccfb6
               Container: chrisstest
                  Object: boot/initrd.img-3.16.0-22-generic
            Content Type: application/octet-stream
          Content Length: 30730085
           Last Modified: Mon, 03 Nov 2014 14:53:52 GMT
                    ETag: 53d3910d4f2cea329b515f19cbb5ca89
Meta X-Object-Meta-Test1: test1
           Accept-Ranges: bytes
             X-Timestamp: 1415026432.67306

Interestingly, "swift download" (and the get_object() method) do re-create the segmented file correctly even without the manifest (does it just make an opportunistic look for chrisstest_segments?)

System details:

$ lsb_release -rd
Description: Ubuntu 14.10
Release: 14.10

$ apt-cache policy python-swiftclient
python-swiftclient:
  Installed: 1:2.3.0-0ubuntu1
  Candidate: 1:2.3.0-0ubuntu1
  Version table:
 *** 1:2.3.0-0ubuntu1 0
        500 http://gb.archive.ubuntu.com/ubuntu/ utopic/main amd64 Packages
        100 /var/lib/dpkg/status

Revision history for this message
Stuart Bishop (stub) wrote :

The swift(1) command line tool is lying.

Revision history for this message
Tim Burke (1-tim-z) wrote :

Looks like POST-as-COPY flattened the DLO; notice how the ETag changed. It worked out reasonably well in this case, although there are some orphaned segments and data duplication. However, as described in bug 1487791, similar POSTs can fail for mysterious (to an end-user) reasons.

Revision history for this message
James Page (james-page) wrote :

Quick test with swift against radosgw:

ubuntu@james-page-bastion:~⟫ swift upload -S 10000000 kerneltest /boot/initrd.img-4.4.0-96-generic
boot/initrd.img-4.4.0-96-generic segment 1
boot/initrd.img-4.4.0-96-generic segment 0
boot/initrd.img-4.4.0-96-generic
ubuntu@james-page-bastion:~⟫ swift stat kerneltest boot/initrd.img-4.4.0-96-generic
       Account: v1
     Container: kerneltest
        Object: boot/initrd.img-4.4.0-96-generic
  Content Type: binary/octet-stream
Content Length: 15604860
 Last Modified: Mon, 30 Oct 2017 11:47:06 GMT
          ETag: "47ecced69b46738eeda009e28c53ce94"
      Manifest: kerneltest_segments/boot/initrd.img-4.4.0-96-generic/1505843367.051525/15604860/10000000/
    Meta Mtime: 1505843367.051525
 Accept-Ranges: bytes
   X-Timestamp: 1509364026.63073
    X-Trans-Id: tx0000000000000000015bc-0059f7114a-2ca27-default
ubuntu@james-page-bastion:~⟫ swift post -m "x-object-meta-test1:test1" kerneltest boot/initrd.img-4.4.0-96-generic
ubuntu@james-page-bastion:~⟫ swift stat kerneltest boot/initrd.img-4.4.0-96-generic
                 Account: v1
               Container: kerneltest
                  Object: boot/initrd.img-4.4.0-96-generic
            Content Type: binary/octet-stream
          Content Length: 15604860
           Last Modified: Mon, 30 Oct 2017 11:47:46 GMT
                    ETag: "47ecced69b46738eeda009e28c53ce94"
                Manifest: kerneltest_segments/boot/initrd.img-4.4.0-96-generic/1505843367.051525/15604860/10000000/
Meta X-Object-Meta-Test1: test1
           Accept-Ranges: bytes
             X-Timestamp: 1509364066.09740
              X-Trans-Id: tx000000000000000000f95-0059f71165-2ca24-default
ubuntu@james-page-bastion:~⟫

LGTM (that's a xenial-ocata based cloud).

checking with swift as well.

Revision history for this message
James Page (james-page) wrote :

And with swift:

ubuntu@james-page-bastion:~/tools/openstack-charm-testing/trunk⟫ swift upload -S 10000000 kerneltest /boot/initrd.img-4.4.0-96-generic boot/initrd.img-4.4.0-96-generic segment 1
boot/initrd.img-4.4.0-96-generic segment 0
boot/initrd.img-4.4.0-96-generic
ubuntu@james-page-bastion:~/tools/openstack-charm-testing/trunk⟫ swift stat kerneltest boot/initrd.img-4.4.0-96-generic
               Account: AUTH_9736b8ef9ac84824ab6e248455665cbf
             Container: kerneltest
                Object: boot/initrd.img-4.4.0-96-generic
          Content Type: application/octet-stream
        Content Length: 15604860
         Last Modified: Mon, 30 Oct 2017 11:49:24 GMT
                  ETag: "47ecced69b46738eeda009e28c53ce94"
              Manifest: kerneltest_segments/boot/initrd.img-4.4.0-96-generic/1505843367.051525/15604860/10000000/
            Meta Mtime: 1505843367.051525
         Accept-Ranges: bytes
           X-Timestamp: 1509364163.49140
            X-Trans-Id: tx07d94f0c87e84717a404c-0059f711c8
X-Openstack-Request-Id: tx07d94f0c87e84717a404c-0059f711c8
ubuntu@james-page-bastion:~/tools/openstack-charm-testing/trunk⟫ swift post -m "x-object-meta-test1:test1" kerneltest boot/initrd.img-4.4.0-96-generic
ubuntu@james-page-bastion:~/tools/openstack-charm-testing/trunk⟫ swift stat kerneltest boot/initrd.img-4.4.0-96-generic
                 Account: AUTH_9736b8ef9ac84824ab6e248455665cbf
               Container: kerneltest
                  Object: boot/initrd.img-4.4.0-96-generic
            Content Type: application/octet-stream
          Content Length: 0
           Last Modified: Mon, 30 Oct 2017 11:49:35 GMT
                    ETag: d41d8cd98f00b204e9800998ecf8427e
Meta X-Object-Meta-Test1: test1
           Accept-Ranges: bytes
             X-Timestamp: 1509364174.89169
              X-Trans-Id: tx006f1ae51d9142be9e46f-0059f711d4
  X-Openstack-Request-Id: tx006f1ae51d9142be9e46f-0059f711d4

As per original bug report.

Revision history for this message
James Page (james-page) wrote :

Looks like this problem persists in swift@ocata

(swift was deployed as part of an ocata cloud).

James Page (james-page)
Changed in swift (Ubuntu):
status: New → Confirmed
importance: Undecided → Low
status: Confirmed → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.