Support partNumber query parameter on object GET
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
In Progress
|
Medium
|
Unassigned | ||
Swift3 |
Invalid
|
Undecided
|
Unassigned |
Bug Description
There's an (undocumented) feature to request individual parts from multipart uploads -- requests look something like
> GET /<bucket>
> Host: s3.amazonaws.com
> User-Agent: curl/7.54.0
> Date: Wed, 29 Nov 2017 20:26:21 +0000
> Authorization: AWS ...
>
< HTTP/1.1 206 Partial Content
< x-amz-id-2: ...
< x-amz-request-id: ...
< Date: Wed, 29 Nov 2017 20:26:22 GMT
< Last-Modified: Sat, 03 Jun 2017 00:40:30 GMT
< ETag: "30ee66ffc6e6d2
< x-amz-mp-
< Accept-Ranges: bytes
< Content-Range: bytes 200000000-
< Content-Type: binary/octet-stream
< Content-Length: 200000000
< Server: AmazonS3
<
This even works with regular objects (note the lack of x-amz-mp-
> GET /<bucket>
> Host: s3-us-west-
> User-Agent: curl/7.54.0
> Date: Wed, 29 Nov 2017 20:28:28 +0000
> Authorization: AWS ...
>
< HTTP/1.1 206 Partial Content
< x-amz-id-2: ...
< x-amz-request-id: ...
< Date: Wed, 29 Nov 2017 20:28:29 GMT
< Last-Modified: Mon, 24 Aug 2015 17:30:47 GMT
< ETag: "60b725f10c9c85
< Accept-Ranges: bytes
< Content-Range: bytes 0-1/2
< Content-Type: binary/octet-stream
< Content-Length: 2
< Server: AmazonS3
<
Asking for a part that doesn't exist (sensibly) 416s:
< HTTP/1.1 416 Requested Range Not Satisfiable
< x-amz-request-id: ...
< x-amz-id-2: ...
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Wed, 29 Nov 2017 20:28:36 GMT
< Server: AmazonS3
<
<?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host s3-us-west-
<Error>
Note that the 1<=partNumber<=1000 restriction is imposed *first*, so you could also get back a 400.
tags: | added: s3api |
tags: | added: slo |
Changed in swift: | |
status: | Confirmed → In Progress |
So the feature seems legit/supported:
https:/ /github. com/aws/ aws-sdk- java/issues/ 1172
The response there suggested just using normal Range requests, but it's nice for parallel download to be able to make requests out directly to the specific segments/disks without overlap. Swift clients can do this today because the manifest and segments themselves are available using:
?multipart- manifest= get
I don't think the API can avoid reading the manifest to find the segment name, but it's an internal request so cheap enough. I think that's a reasonable approach for s3api/swift3 to support the API - and it'd be nice to do if there's some sdk's that will let applications take advantage of it easily.