OpenStack Core Infrastructure

swift: no space left on device

Bug #1358779 reported by Felipe Reyes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Core Infrastructure
Incomplete
Undecided
Unassigned
devstack
Invalid
Undecided
Unassigned

Bug Description

logstash query: message:"No space left on device" AND filename:logs*screen-s-*.txt

http://logs.openstack.org/94/108394/3/check/check-tempest-dsvm-neutron-full/4b574bb/logs/screen-s-proxy.txt.gz

proxy-server: ERROR 500 Traceback (most recent call last):#012 File "/opt/stack/new/swift/swift/obj/server.py", line 690, in __call__#012 res = method(req)#012 File "/opt/stack/new/swift/swift/common/utils.py", line 2448, in wrapped#012 return func(*a, **kw)#012 File "/opt/stack/new/swift/swift/common/utils.py", line 1033, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/opt/stack/new/swift/swift/obj/server.py", line 432, in PUT#012 upload_size = writer.write(chunk)#012 File "/opt/stack/new/swift/swift/obj/diskfile.py", line 754, in write#012 self._threadpool.run_in_thread(_write_entire_chunk, chunk)#012 File "/opt/stack/new/swift/swift/common/utils.py", line 2810, in run_in_thread#012 result = func(*args, **kwargs)#012 File "/opt/stack/new/swift/swift/obj/diskfile.py", line 750, in _write_entire_chunk#012 written = os.write(self._fd, chunk)#012OSError: [Errno 28] No space left on device#012 From Object Server re: /v1/AUTH_1770576bc52c4a70a1d70d55a106566f/glance/82feff21-5efe-4601-8c97-6553813af605 127.0.0.1:6013/sdb1 (txn: tx3f78ac01d0b04742822b3-0053f1aead)
proxy-server: Object PUT returning 503 for [500] (txn: tx3f78ac01d0b04742822b3-0053f1aead) (client_ip: 127.0.0.1)

Tags: swift
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/115306

Revision history for this message
Clark Boylan (cboylan) wrote :

Marking this as incomplete for infra and adding devstack to this bug because devstack is creating the swift loopback disk that is used in this job. It appears to be 1G in size if tempest is not enabled and 6G in size if tempest is enabled.

If 6G is too small we should increase the size in devstack so that tempest continues to work for everyone (instead of modifying the default value in devstack-gate for example).

The code that needs updating is at https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/swift#n64.

Changed in openstack-ci:
status: New → Incomplete
Revision history for this message
Matt Riedemann (mriedem) wrote :

The query in the related elastic-recheck query isn't showing any hits anymore so this is probably fixed by something else by now, maybe when we reduced the number of workers, i.e. since we used to have `ncpu` glance workers which could have been doing a lot of work with swift.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on elastic-recheck (master)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/115306
Reason: This isn't hitting anymore and the bug is marked as incomplete.

Revision history for this message
Sean Dague (sdague) wrote :

This devstack bug was last updated over 180 days ago, as devstack
is a fast moving project and we'd like to get the tracker down to
currently actionable bugs, this is getting marked as Invalid. If the
issue still exists, please feel free to reopen it.

Changed in devstack:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.