snapshot dump timeout with /var/log/ - 16G

Python teams. such bad UX issues should have High priority.

Workaround

while true; do if [ -z "$(ps aux | grep -v grep | grep xz)" ] ; then echo "Wating..."; sleep 10; else tar czvf /root/$(ls /var/log/dump/ | grep -v xz).tar.gz /var/log/dump/$(ls /var/log/dump/ | grep -v xz) ; exit; fi ; done

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

Reproduced with 9.0-188
3 controllers, 20 computes+ceph, 179 computes
[root@fuel ~]# du -hs /var/dump/*
44G /var/dump/fuel-snapshot-2016-04-12_08-07-14

The first, most simple, solution would be just to increase timeout in astute.
The second solution is more complex. We may offload compressing task to nodes. This will save some time on transferring and a lot of space on master node. Part of this job can be offloaded, for instance, to logrotate.

The second solution can be implemented in bounds of https://blueprints.launchpad.net/fuel/+spec/manage-logs-with-free-space-consideration

A little test
Uncompressed data size is 15G
1. Creating tar.xz. Resulting archive: 5.4Gb, time consumed 1h 20min.
1. Creating tar.gz. Resulting archive: 7.6Gb, time consumed 8min!!!!
3. Creating tar.bz2. Resulting archive: 7Gb, time consumed 42min

This benchmark http://catchchallenger.first-world.info//wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO seems to confirm my results. It also shows that xz is a memory hog comparing to gz which is veery humble in this sense.

I think that switching back to gz would be a reasonable choice for the simple solution.

Fix proposed to branch: master
Review: https://review.openstack.org/310446

Fix proposed to branch: master
Review: https://review.openstack.org/320345

We are going to deprecate Shotgun and we are not going to fix the problem completely. The fix is just a mitigation. It just improves compression speed so that we encounter timeouts a bit more rarely.

Issue is still reproduced on large scale

There always be some size of a data which would exceed any timeout we set.

Reviewed: https://review.openstack.org/320345
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=18308a743c71e5b24efaf162bac5bbd2f7290e78
Submitter: Jenkins
Branch: master

commit 18308a743c71e5b24efaf162bac5bbd2f7290e78
Author: Georgy Kibardin <email address hidden>
Date: Tue May 24 13:30:42 2016 +0300

    Use gzip for snapshots

    Shotgun has been changed to use gzip for snapshot.
    On the 15G snapshot data gz is 10 times faster than xz while compression
    rate is just 36% against 50% with gzip.
    For the pattern "create once - download once" this looks like a
    reasonable solution.

    Change-Id: I133ae854c619655169f6b42003087dd9cc21b8e0
    Closes-Bug: #1546023

https://review.openstack.org/#/c/320345/ should be backported to 9.0

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/328199

Reviewed: https://review.openstack.org/328199
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=9f02eb988bf732ece0f4bad19142a4a6834f7c5b
Submitter: Jenkins
Branch: stable/mitaka

commit 9f02eb988bf732ece0f4bad19142a4a6834f7c5b
Author: Georgy Kibardin <email address hidden>
Date: Tue May 24 13:30:42 2016 +0300

    Use gzip for snapshots

    Shotgun has been changed to use gzip for snapshot.
    On the 15G snapshot data gz is 10 times faster than xz while compression
    rate is just 36% against 50% with gzip.
    For the pattern "create once - download once" this looks like a
    reasonable solution.

    Change-Id: I133ae854c619655169f6b42003087dd9cc21b8e0
    Closes-Bug: #1546023
    (cherry picked from commit 18308a743c71e5b24efaf162bac5bbd2f7290e78)

I put in Confirmed state because the issue is still reproduced on 200 nodes.

The bug is not fixed since it is impossible, there always be some amount of nodes which would exceed any timeout we set. The issue has just been mitigated not completely fixed.