Mirantis OpenStack

'openstack object create' and 'swift upload' can not upload big (5432MB) file to container.

  1. Series 10.0.x
  2. Bug #1583033
Bug #1583033 reported by Alexander Koryagin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
Confirmed
Wishlist
MOS Oslo
Mirantis OpenStack 10.0
9.x
Won't Fix
Wishlist
MOS Oslo
Mirantis OpenStack 9.0

Bug Description

Hello,
Please take a look at the issue below:
    'openstack object create' and 'swift upload' can not upload big (5432MB) file to container.

Error messages:
 - Unable to establish connection to http://10.109.4.6:8080/swift/v1/TESTBUCKET01//tmp/ObjStor_somefile
 - ('Connection aborted.', error(32, 'Broken pipe'))

My env: MOS 9.0 ISO# 327.

Actions:
1) OK - Create file:
  # fallocate -l 5432M /tmp/ObjStor_somefile
  # ls -alph /tmp/ObjStor_somefile
      -rw-r--r-- 1 root root 5.4G May 18 07:41 /tmp/ObjStor_somefile

2) NOK - With a help of 'openstack' try to put file to container:
  # . openrc
  # openstack container create TESTBUCKET01 -f json
      [
        {
          "account": "v1",
          "container": "TESTBUCKET01",
          "x-trans-id": "tx00000000000000000065e-00573c1cd0-5ea5-default"
        }
      ]

  # openstack object create TESTBUCKET01 /tmp/ObjStor_somefile -f json
      >>> Unable to establish connection to http://10.109.4.6:8080/swift/v1/TESTBUCKET01//tmp/ObjStor_somefile

  # openstack object list TESTBUCKET01 --long -f json
      []

3) NOK - With a help of swift try to upload big file to container:
  # swift post TESTBUCKET02
  # swift upload TESTBUCKET02 /tmp/ObjStor_somefile
      >>> ('Connection aborted.', error(32, 'Broken pipe'))

  # swift list -l TESTBUCKET02
      {empty}

4) OK - With a help of swift with 'segment' option try to upload big file to container:
  # swift post TESTBUCKET03
  # swift upload TESTBUCKET03 -S 1024M /tmp/ObjStor_somefile
      tmp/ObjStor_somefile segment 5
      tmp/ObjStor_somefile segment 3
      tmp/ObjStor_somefile segment 1
      tmp/ObjStor_somefile segment 4
      tmp/ObjStor_somefile segment 2
      tmp/ObjStor_somefile segment 0
      tmp/ObjStor_somefile

  # swift list -l TESTBUCKET03
             0 2016-05-18 08:01:53 None tmp/ObjStor_somefile
             0

  # swift list -l TESTBUCKET03_segments
    1073741824 2016-05-18 08:01:52 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000000
    1073741824 2016-05-18 08:01:47 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000001
    1073741824 2016-05-18 08:01:49 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000002
    1073741824 2016-05-18 08:01:42 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000003
    1073741824 2016-05-18 08:01:49 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000004
     327155712 2016-05-18 07:54:50 None tmp/ObjStor_somefile/1463557270.178049/5695864832/1073741824/00000005
    5695864832

Revision history for this message
Alexander Koryagin (akoryagin) wrote :
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Swift team, please take a look.

Changed in mos:
assignee: nobody → Alyona Kiseleva (akiselyova)
status: New → Confirmed
importance: Undecided → High
milestone: none → 9.0
tags: added: area-swift
Revision history for this message
Alyona Kiseleva (akiselyova) wrote :

Swift maximum file size is 5GB and if you want to upload bigger files to this storage, it has to be segmented, it's expected behavior according documentation (see http://docs.openstack.org/developer/swift/overview_large_objects.html, "Swift has a limit on the size of a single uploaded object; by default this is 5GB").
So, use swift option -S is decision.

What about "openstack object create" command, in fact I don't know, how it works inside, but if it uses something like swift upload, it could check file size and use segmented upload in case of big files. But I'm not sure, that it's supported feature in openstack cli, no info about large file support in manual to this command. It's question to somebody, more familiar with internal work of openstack cli.

Changed in mos:
status: Confirmed → Invalid
Revision history for this message
Alexander Koryagin (akoryagin) wrote :

Hello,

So we have a clear error here:
. . .
2) NOK - With a help of 'openstack' try to put file to container:
  # . openrc
  # openstack container create TESTBUCKET01 -f json
      [
        {
          "account": "v1",
          "container": "TESTBUCKET01",
          "x-trans-id": "tx00000000000000000065e-00573c1cd0-5ea5-default"
        }
      ]

  # openstack object create TESTBUCKET01 /tmp/ObjStor_somefile -f json
      >>> Unable to establish connection to http://10.109.4.6:8080/swift/v1/TESTBUCKET01//tmp/ObjStor_somefile

  # openstack object list TESTBUCKET01 --long -f json
      []
. . .
And there is no option for segmentation in 'openstack object create'.

Why do you think that this issue is 'invalid'?

Alexander Koryagin (akoryagin)
Changed in mos:
status: Invalid → New
Revision history for this message
Alyona Kiseleva (akiselyova) wrote :

Ok, in that case it needs to be reassigned, I'm not a specialist in openstack cli code.

Changed in mos:
assignee: Alyona Kiseleva (akiselyova) → nobody
status: New → Confirmed
Revision history for this message
Dina Belova (dbelova) wrote :

Assigning to mos-oslo team. Please take a look and say if you're the right people to work through this.

Changed in mos:
assignee: nobody → MOS Oslo (mos-oslo)
Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

Dina, probably we are, but I would rather question importance of that issue. This is definitely not a high bug - there is a clear solution - use swift client. Moreover, this is more like a feature request for openstackclient, not a bug at all.

Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

Agreed with Dina that that is a feature request, hence moving to wishlist priority and assigning to 10.0.

Changed in mos:
status: Confirmed → Won't Fix
Dmitry Mescheryakov (dmitrymex)
tags: added: 10.0-reviewed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.