Fuel for OpenStack

Cannot perform backup with dockerctl backup /path/to/backup

Bug #1544204 reported by Artem Hrechanychenko on 2016-02-10

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Invalid	High	Matthew Mosesohn	Fuel for OpenStack 10.0
8.0.x	Fix Released	High	Anton Chevychalov	Fuel for OpenStack 8.0-mu-3
Mitaka	Invalid	High	Matthew Mosesohn	Fuel for OpenStack 9.0

Bug Description

steps to reproduce:

1)create volume - 50GB
2)mount as - /var/log/backup_dir/
2)perform dockerctl backup to non-default path

Expected result:
Backup procedure started

Actual result:
[root@nailgun ~]# dockerctl backup /var/log/backup_dir/
df: ‘/var/backup/fuel’: No such file or directory
Insufficient disk space to perform backup. At least 2gb must be free on /var/backup/fuel partition.
Backup failed!

Additional information:
Custom backup destination should perform a `df` check only on the custom backup directory. /var/backup/fuel shouldn't be part of the equation.

The backup is forced to fullbackup mode by default, which takes an enormous amount of time. Minimal backup was introduced in 7.0, but no longer works. This is a regression in backup capabilities.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

Snapshot is not available due to exhausted space in /var directory.

See original description

Tags:

Artem Hrechanychenko (agrechanichenko) on 2016-02-10

description:

updated

Artem Hrechanychenko (agrechanichenko) on 2016-02-10

Changed in fuel:
assignee:	nobody → Matthew Mosesohn (raytrac3r)

Ilya Kutukov (ikutukov) on 2016-02-11

Changed in fuel:
status:	New → Confirmed
tags:	added: area-library

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2016-02-24:

This is a regression introduced in https://review.openstack.org/#/c/273037/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-02-24: Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/284263

Matthew Mosesohn (raytrac3r) on 2016-03-02

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2016-03-18: Re: cannot perfrom backup with dockerctl backup /path/to/backup

Waiting on 8.0mu1 approval

Changed in fuel:
importance:	Medium → High

Vitaly Sedelnik (vsedelnik) on 2016-03-25

Changed in fuel:
milestone:	8.0-mu-1 → 8.0-updates

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2016-04-08:

This commit should not stay unmerged indefinitely. It really has a high priority status for a good reason. Without merging https://review.openstack.org/#/c/284263/ , a user cannot do the following:
* back up just ephemeral data (only full docker backup is enabled)
* back up to a custom location (disk space check is broken)

Matthew Mosesohn (raytrac3r) on 2016-04-18

summary:	- cannot perfrom backup with dockerctl backup /path/to/backup + Cannot perform backup with dockerctl backup /path/to/backup
description:	updated

Vitaly Sedelnik (vsedelnik) on 2016-04-27

Changed in fuel:
milestone:	8.0-updates → 8.0-mu-2

Fuel Devops McRobotson (fuel-devops-robot) on 2016-05-06

Changed in fuel:
milestone:	8.0-mu-2 → 10.0
status:	In Progress → Confirmed

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2016-05-06:

This is invalid for 9.0 and 10.0 because there is no dockerctl

Changed in fuel:
status:	Confirmed → Invalid

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-19: Fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/284263
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e7d0ab4f8b47bf4bb194836e3f147c609638b8dc
Submitter: Jenkins
Branch: stable/8.0

commit e7d0ab4f8b47bf4bb194836e3f147c609638b8dc
Author: Matthew Mosesohn <email address hidden>
Date: Wed Feb 24 20:06:05 2016 +0300

Fix disk space calculation for custom backup locations

    This fixes a regression introduced by
    4207659be2a20dbd0c3bd0f20dc50751694b0061 which broke behavior
    where disk space checking was skipped for non-default locations.

Also fixes 02393e503db32ee3c790e249df11c967dfcd1af4 which
accidentally forced full backups.

Change-Id: I44174bf7f9aca2d4242a275a7dcedc4545489e8f
Closes-Bug: #1544204

Dmitry Belyaninov (dbelyaninov) on 2016-06-23

tags:

added: on-verification

Revision history for this message

Dmitry Belyaninov (dbelyaninov) wrote on 2016-06-24:

8.0 :

[root@nailgun ~]# dockerctl backup /var/log/backup_dir/
df: вЂ/var/backup/fuelвЂ™: No such file or directory
Insufficient disk space to perform backup. At least 2gb must be free on /var/backup/fuel partition.
Backup failed!
[root@nailgun ~]#

8.0 MU2 :

[root@nailgun ~]# dockerctl backup /var/log/backup_dir
Pausing containers...
....
Backing up PostgreSQL database to /var/log/backup_dir/postgres_backup.sql...
Compressing archives...
/var/log/backup_dir/fuel_backup_2016-06-23_1237.tar - Compression Ratio: 30.564. Average Compression Speed: 38.000MB/s.
Total time: 00:00:02.29
Cleaning up...
Backup complete. File is available at /var/log/backup_dir/fuel_backup_2016-06-23_1237.tar.lrz
[root@nailgun ~]#

Revision history for this message

Denis Puchkin (dpuchkin) wrote on 2016-07-06:

Minimal backup does not work correctly for postgres data backup, after restore from "minimal backup" the postgres database may be broken

minimal backup script incorrectly creates a backup copy of the database:
1. pause postgres container
2. tar postgress data folder (in this step we have inconsistent data)
3. unpause container

restore:
1. stop postgress conteiner
2. untar data folder
3. start container (in this step, postgres try to recover data after not properly shut down, but sometimes unsuccessfully)

as a result, we have broken fuel

logs:
backup/restore https://paste.mirantis.net/show/2446/
postgers log: https://paste.mirantis.net/show/2447/
failed swarm test: https://patching-ci.infra.mirantis.net/view/8.0.swarm/job/8.0.system_test.ubuntu.backup_restore_master/24/

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2016-07-06:

Reverting the original fix, reopening the bug and targeting it to the next MU. We need better fix which stops postgres db correctly.

Revision history for this message

Denis Puchkin (dpuchkin) wrote on 2016-07-06:

#10

Revert CR https://review.openstack.org/#/c/338212/

Revision history for this message

Denis Puchkin (dpuchkin) wrote on 2016-07-06:

#11

right CR link https://review.openstack.org/#/c/338185/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-08-31: Fix proposed to fuel-library (stable/8.0)

#12

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/363774

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-02: Fix merged to fuel-library (stable/8.0)

#13

Reviewed: https://review.openstack.org/363774
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=334f764de8839d26d97be75489dbb0cfe6a8395f
Submitter: Jenkins
Branch: stable/8.0

commit 334f764de8839d26d97be75489dbb0cfe6a8395f
Author: Anton Chevychalov <email address hidden>
Date: Wed Aug 31 18:54:12 2016 +0300

Calculate free space for custom backup dir

Fix verify_disk_space procedure to check custom
backup dir provided in command line.

    Fix regression with restore database.
    We have no proper way to restore database during
    nonfull backup because nonfull backup was masked
    by error in verify_disk_space function.
    Full restore was also switched to proper restore
    of database to increase stability of restoring.
    Also fix compress and uncompress to more efficient way.

Change-Id: Idd0bead1d8362511be4d9550f7fab0f5c88a7724
Closes-Bug: #1544204

Revision history for this message

Dmitry (dtsapikov) wrote on 2016-09-14:

#14

Reproduced on 8.0+mu2

[root@fuel ~]# dockerctl backup /var/log/backup_dir/
df: «/var/backup/fuel»: Нет такого файла или каталога
Insufficient disk space to perform backup. At least 2gb must be free on /var/backup/fuel partition.
Backup failed!

Verified on 8.0+mu3

[root@fuel ~]# dockerctl backup /var/log/backup_dir/
Checking space in /var/log/backup_dir/ for backup
Pausing containers...
206e004c6c9d
88687dfb5120
ec994dc69488
54dec0c19f70
add12468cc33
9e02c060398f
bb16ef892df3
d1ba755b6f58
a23432ceb64c
904341608a76
670289e816ce
Archiving system folders
tar: Удаляется начальный `/' из имен объектов
Unpausing containers...
206e004c6c9d
88687dfb5120
ec994dc69488
54dec0c19f70
add12468cc33
9e02c060398f
bb16ef892df3
d1ba755b6f58
a23432ceb64c
904341608a76
670289e816ce
Backing up PostgreSQL database to /var/log/backup_dir//postgres_backup.sql...
Compressing archives...
Compressing system-dirs.tar postgres_backup.sql
Compression Ratio: 28.883. Average Compression Speed: 25.667MB/s.
Total time: 00:00:03.53
Cleaning up...
Backup complete. File is available at /var/log/backup_dir//fuel_backup_2016-09-14_1355.tar.lrz
[root@fuel ~]#
[root@fuel ~]#
[root@fuel ~]# cd /var/log/backup_dir/
[root@fuel backup_dir]# ls
fuel_backup_2016-09-14_1355.tar.lrz lost+found
[root@fuel backup_dir]#

tags:

removed: on-verification

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.