Mirantis OpenStack

Timmy exit code: 100 in case when you've launched it from UI

Bug #1651789 reported by Pavel on 2016-12-21

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Mirantis OpenStack	Confirmed	Medium	Fuel Sustaining	Mirantis OpenStack 9.2

Bug Description

Hello.

Timmy return "Timmy exit code: 100"

Steps to reproduce:
1. Deploy 200 env cluster.
2. Wait 3 days.
3. Click on "create diagnostic snapshot" in UI. I expect that my logs will be collected without any errors.
4. Got "You have not enough free space" error because of
>> 2016-12-21 13:59:11,540 INFO: nodes: logs_populate(): fuel: total logs size: 123835MB
(123 GB for last 3 days)

Maybe, it would be good to add "--logs-coeff" parameter here
https://github.com/openstack/fuel-astute/blob/master/lib/astute/dump.rb#L38 ?

Tags:

Revision history for this message

Pavel (pshvetsov) wrote on 2016-12-21:

timmy exit code: 100 Edit (65.0 KiB, image/png)

Also, take a look on attached picture.
I suppose we should print more information here.

Revision history for this message

Dmitry Sutyagin (dsutyagin) wrote on 2016-12-21:

@Pavel

This is not a bug in Timmy. For a big cluster you need a lot of space for the snapshot. I agree that logs-coeff could help BUT the problem is that current astute wrapper for Timmy repacks all collected data into a tar archive, doubling the space required. Therefore using any <1 logs coefficient may result in out of space issues, considering that default compression ratio is around 0.3, imagine it going up to 0.5 and that's it, we'll run out of space.

Re-assigning to Georgy Kibardin.

My suggestion is to expand UI for the snapshot:
1.1 - allow checking for required space separately from collecting logs
1.2 - allow selecting from which nodes to collect snapshot (to decrease space requirements)
1.3 - stop generating a tar with all collected files, provide links to individual files, then --logs-coeff can be more or less safely used with a value of, say, 0.6.
1.4 - allow user to set --logs-coeff from within the UI (with a warning that setting to low will result in out of space situation, and values below 0.5 are not recommended).

Changed in mos:
assignee:	Alexander Dobdin (sanek-dobdin) → Georgy Kibardin (gkibardin)

Revision history for this message

Dmitry Sutyagin (dsutyagin) wrote on 2016-12-21:

1.5 - allow excluding atop logs, which may decrease space requirements significantly (up to 2x)

Revision history for this message

Georgy Kibardin (gkibardin) wrote on 2016-12-21:

Pavel, could you please provide "df -h" output from master node?

Changed in mos:
status:	New → Incomplete

Revision history for this message

Pavel (pshvetsov) wrote on 2016-12-22:

Georgy,

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.5G 3.9G 5.2G 43% /
devtmpfs 9.8G 0 9.8G 0% /dev
tmpfs 9.8G 0 9.8G 0% /dev/shm
tmpfs 9.8G 193M 9.6G 2% /run
tmpfs 9.8G 0 9.8G 0% /sys/fs/cgroup
/dev/mapper/os-var 185G 19G 158G 11% /var
/dev/vda3 197M 158M 39M 81% /boot
/dev/mapper/os-varlog 278G 124G 140G 47% /var/log
/dev/vda2 200M 0 200M 0% /boot/efi
tmpfs 2.0G 0 2.0G 0% /run/user/0

Revision history for this message

Georgy Kibardin (gkibardin) wrote on 2016-12-22:

Timmy UI integration is not perfect yet and requires twice as mach space to gather and publish a snapshot for downloading. For huge files it is recommended to use cli - it saves time and space excluding master node mediation. It also gives a lot more fine grained control over what is downloaded which is important when a snapshot is too big.