Activity log for bug #1810026

Date Who What changed Old value New value Message
2018-12-29 09:54:53 Bhaskar Singhal bug added bug
2018-12-29 10:01:44 Bhaskar Singhal description I am trying to upload an object (100+MB) to swift3/swift using the AWS .Net SDK multi-segment upload ( https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html ). But the upload keeps failing with 'Amazon.Runtime.AmazonClientException' Expected hash not equal to calculated hash. This is happening because the ETag calculated in obj/server is passed on the entire chunk, but in this case, the chunk has the chunk-signature etc. We need to get the decoded content and then calculate ETag. https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html Various headers set in the request: {'HTTP_AUTHORIZATION': 'AWS4-HMAC-SHA256 Credential=4a203e8a63cc4fdfa4bbd3ea6c208bac/20181228/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=e7d32398ec602bd2fbfbbcd613ba26835d2036f6215862255a04ee91dadb242e', 'SCRIPT_NAME': '', 'keystone.token_auth': <keystonemiddleware.auth_token._user_plugin.UserAuthPlugin object at 0x7f00e68d5b90>, 'webob.adhoc_attrs': {'response': <_AuthTokenResponse at 0x7f00e68d5f50 200 OK>}, 'REQUEST_METHOD': 'PUT', 'HTTP_X_AMZ_DATE': '20181228T181903Z', 'PATH_INFO': '/bucket1/BBBunny123', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': 'partNumber=4&uploadId=YzQ0Yjk2MzMtMzdhYy00NDkyLWEwZjItZjBhYjA0M2FjYjMx', 'REMOTE_ADDR': '150.50.2.89', 'CONTENT_LENGTH': '5246566', 'HTTP_X_AMZ_DECODED_CONTENT_LENGTH': '5242880', 'HTTP_USER_AGENT': 'aws-sdk-dotnet-45/3.3.8.2 aws-sdk-dotnet-core/3.3.17.0 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_6.2.9200.0 ClientAsync TransferManager/MultipartUploadCommand', 'eventlet.posthooks': [], 'RAW_PATH_INFO': '/bucket1/BBBunny123', 'REMOTE_PORT': '58993', 'eventlet.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_X_IDENTITY_STATUS': 'Invalid', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '80', 'HTTP_X_AMZ_CONTENT_SHA256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'wsgi.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_HOST': 'vsa-00000ad7-zadara-dev2.zadarazios.com', 'wsgi.multithread': True, 'HTTP_EXPECT': '100-continue', 'wsgi.version': (1, 0), 'SERVER_NAME': '150.50.2.132', 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x7f00df2665d0>, 'wsgi.multiprocess': False, 'swift.trans_id': 'tx8f4b1c9093d1422389664-005c266917', 'CONTENT_TYPE': 'text/plain', 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x7f00df2667d0>} I am trying to upload an object (100+MB) to swift3/swift using the AWS .Net SDK multi-segment upload ( https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html ). But the upload keeps failing with 'Amazon.Runtime.AmazonClientException' Expected hash not equal to calculated hash. This is happening because the ETag calculated in obj/server is passed on the entire chunk, but in this case, the chunk has the chunk-signature etc. We need to get the decoded content and then calculate ETag. https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html Various headers set in the request: {'HTTP_AUTHORIZATION': 'AWS4-HMAC-SHA256 Credential=4a203e8a63cc4fdfa4bbd3ea6c208bac/20181228/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=e7d32398ec602bd2fbfbbcd613ba26835d2036f6215862255a04ee91dadb242e', 'SCRIPT_NAME': '', 'keystone.token_auth': <keystonemiddleware.auth_token._user_plugin.UserAuthPlugin object at 0x7f00e68d5b90>, 'webob.adhoc_attrs': {'response': <_AuthTokenResponse at 0x7f00e68d5f50 200 OK>}, 'REQUEST_METHOD': 'PUT', 'HTTP_X_AMZ_DATE': '20181228T181903Z', 'PATH_INFO': '/bucket1/BBBunny123', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': 'partNumber=4&uploadId=YzQ0Yjk2MzMtMzdhYy00NDkyLWEwZjItZjBhYjA0M2FjYjMx', 'REMOTE_ADDR': '150.50.2.89', 'CONTENT_LENGTH': '5246566', 'HTTP_X_AMZ_DECODED_CONTENT_LENGTH': '5242880', 'HTTP_USER_AGENT': 'aws-sdk-dotnet-45/3.3.8.2 aws-sdk-dotnet-core/3.3.17.0 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_6.2.9200.0 ClientAsync TransferManager/MultipartUploadCommand', 'eventlet.posthooks': [], 'RAW_PATH_INFO': '/bucket1/BBBunny123', 'REMOTE_PORT': '58993', 'eventlet.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_X_IDENTITY_STATUS': 'Invalid', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '80', 'HTTP_X_AMZ_CONTENT_SHA256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'wsgi.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_HOST': 'hostname.xxx.xxx.xxx', 'wsgi.multithread': True, 'HTTP_EXPECT': '100-continue', 'wsgi.version': (1, 0), 'SERVER_NAME': '150.50.2.132', 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x7f00df2665d0>, 'wsgi.multiprocess': False, 'swift.trans_id': 'tx8f4b1c9093d1422389664-005c266917', 'CONTENT_TYPE': 'text/plain', 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x7f00df2667d0>} Thanks, Bhaskar
2018-12-29 10:05:48 Bhaskar Singhal description I am trying to upload an object (100+MB) to swift3/swift using the AWS .Net SDK multi-segment upload ( https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html ). But the upload keeps failing with 'Amazon.Runtime.AmazonClientException' Expected hash not equal to calculated hash. This is happening because the ETag calculated in obj/server is passed on the entire chunk, but in this case, the chunk has the chunk-signature etc. We need to get the decoded content and then calculate ETag. https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html Various headers set in the request: {'HTTP_AUTHORIZATION': 'AWS4-HMAC-SHA256 Credential=4a203e8a63cc4fdfa4bbd3ea6c208bac/20181228/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=e7d32398ec602bd2fbfbbcd613ba26835d2036f6215862255a04ee91dadb242e', 'SCRIPT_NAME': '', 'keystone.token_auth': <keystonemiddleware.auth_token._user_plugin.UserAuthPlugin object at 0x7f00e68d5b90>, 'webob.adhoc_attrs': {'response': <_AuthTokenResponse at 0x7f00e68d5f50 200 OK>}, 'REQUEST_METHOD': 'PUT', 'HTTP_X_AMZ_DATE': '20181228T181903Z', 'PATH_INFO': '/bucket1/BBBunny123', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': 'partNumber=4&uploadId=YzQ0Yjk2MzMtMzdhYy00NDkyLWEwZjItZjBhYjA0M2FjYjMx', 'REMOTE_ADDR': '150.50.2.89', 'CONTENT_LENGTH': '5246566', 'HTTP_X_AMZ_DECODED_CONTENT_LENGTH': '5242880', 'HTTP_USER_AGENT': 'aws-sdk-dotnet-45/3.3.8.2 aws-sdk-dotnet-core/3.3.17.0 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_6.2.9200.0 ClientAsync TransferManager/MultipartUploadCommand', 'eventlet.posthooks': [], 'RAW_PATH_INFO': '/bucket1/BBBunny123', 'REMOTE_PORT': '58993', 'eventlet.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_X_IDENTITY_STATUS': 'Invalid', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '80', 'HTTP_X_AMZ_CONTENT_SHA256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'wsgi.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_HOST': 'hostname.xxx.xxx.xxx', 'wsgi.multithread': True, 'HTTP_EXPECT': '100-continue', 'wsgi.version': (1, 0), 'SERVER_NAME': '150.50.2.132', 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x7f00df2665d0>, 'wsgi.multiprocess': False, 'swift.trans_id': 'tx8f4b1c9093d1422389664-005c266917', 'CONTENT_TYPE': 'text/plain', 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x7f00df2667d0>} Thanks, Bhaskar I am trying to upload an object (100+MB) to swift3/swift using the AWS .Net SDK multi-segment upload ( https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileDotNet.html ). But the upload keeps failing with 'Amazon.Runtime.AmazonClientException' Expected hash not equal to calculated hash. This is happening because the ETag calculated in obj/server is on the entire chunk, but in this case, the chunk has the chunk-signature etc. We need to decoded the chunk and then calculate ETag. https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html for chunk in iter(timeout_reader, ''): start_time = time.time() if start_time > upload_expiration: self.logger.increment('PUT.timeouts') return HTTPRequestTimeout(request=request) etag.update(chunk) --> this chunk has signature, etc upload_size = writer.write(chunk) elapsed_time += time.time() - start_time Various headers set in the request: {'HTTP_AUTHORIZATION': 'AWS4-HMAC-SHA256 Credential=4a203e8a63cc4fdfa4bbd3ea6c208bac/20181228/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=e7d32398ec602bd2fbfbbcd613ba26835d2036f6215862255a04ee91dadb242e', 'SCRIPT_NAME': '', 'keystone.token_auth': <keystonemiddleware.auth_token._user_plugin.UserAuthPlugin object at 0x7f00e68d5b90>, 'webob.adhoc_attrs': {'response': <_AuthTokenResponse at 0x7f00e68d5f50 200 OK>}, 'REQUEST_METHOD': 'PUT', 'HTTP_X_AMZ_DATE': '20181228T181903Z', 'PATH_INFO': '/bucket1/BBBunny123', 'SERVER_PROTOCOL': 'HTTP/1.0', 'QUERY_STRING': 'partNumber=4&uploadId=YzQ0Yjk2MzMtMzdhYy00NDkyLWEwZjItZjBhYjA0M2FjYjMx', 'REMOTE_ADDR': '150.50.2.89', 'CONTENT_LENGTH': '5246566', 'HTTP_X_AMZ_DECODED_CONTENT_LENGTH': '5242880', 'HTTP_USER_AGENT': 'aws-sdk-dotnet-45/3.3.8.2 aws-sdk-dotnet-core/3.3.17.0 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_6.2.9200.0 ClientAsync TransferManager/MultipartUploadCommand', 'eventlet.posthooks': [], 'RAW_PATH_INFO': '/bucket1/BBBunny123', 'REMOTE_PORT': '58993', 'eventlet.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_X_IDENTITY_STATUS': 'Invalid', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '80', 'HTTP_X_AMZ_CONTENT_SHA256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'wsgi.input': <eventlet.wsgi.Input object at 0x7f00df266250>, 'HTTP_HOST': 'hostname.xxx.xxx.xxx', 'wsgi.multithread': True, 'HTTP_EXPECT': '100-continue', 'wsgi.version': (1, 0), 'SERVER_NAME': '150.50.2.132', 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x7f00df2665d0>, 'wsgi.multiprocess': False, 'swift.trans_id': 'tx8f4b1c9093d1422389664-005c266917', 'CONTENT_TYPE': 'text/plain', 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x7f00df2667d0>} Thanks, Bhaskar
2019-01-03 19:13:06 Tim Burke swift: status New Confirmed
2019-01-03 19:13:11 Tim Burke swift: importance Undecided Low
2019-01-03 19:13:25 Tim Burke tags object-server s3api slo s3api
2019-01-04 04:16:35 Bhaskar Singhal bug watch added https://github.com/aws/aws-sdk-net/issues/678
2019-02-15 13:50:00 OpenStack Infra tags s3api in-feature-losf s3api
2019-12-31 03:19:52 fengli bug added subscriber fengli
2020-10-21 20:06:57 clayg swift: importance Low Medium
2022-04-06 09:20:04 OpenStack Infra swift: status Confirmed In Progress