Temporary files from manually specified snapshot directory use default dir

My 2 cents:

[root@fuel ~]# fuel snapshot --dir /var/www/nailgun/dump/snapshot_1
Generating dump...
Downloading: http://10.20.7.2:8000/dump/fuel-snapshot-2015-02-10_01-21-57.tgz Bytes: 253419832
[==============================================================================]()
[root@fuel ~]# fuel snapshot
Generating dump...
Downloading: http://10.20.7.2:8000/dump/fuel-snapshot-2015-02-10_02-07-25.tgz Bytes: 253894455
[==============================================================================]()

[root@fuel dump]# ls -la /var/www/nailgun/dump/
total 495436
drwxr-xr-x 3 root root 122 Feb 10 02:47 .
drwxr-xr-x. 7 root root 4096 Jan 4 06:54 ..
-rw-r--r-- 1 root root 253419832 Feb 10 02:02 fuel-snapshot-2015-02-10_01-21-57.tgz
-rw-r--r-- 1 root root 253894455 Feb 10 02:47 fuel-snapshot-2015-02-10_02-07-25.tgz
-rw-r--r-- 1 root root 59 Feb 10 02:47 last
drwxr-xr-x 2 root root 50 Feb 10 02:02 snapshot_1

After creation of two snapshots both of them have a copy stored in /var/www/nailgun/dump/.
Is it by mistake, or design, that we store all previous snapshots without rotation somewhere in the filesystem, occupying space, and not talking to the user about that? To figure out the directory one should check the source code.

Directory parameter is not even passed to API call, and there is a couple of resons for this, it is only used to download snapshot on
your machine in specified place.

First of all fuelclient should be usable not only from master node, there is couple of exceptions, but they are considered as bugs, that we will fix asap.
2nd - snapshot generated at /var/www because at that location it is served by nginx.

So, we cant know that it is expected to delete snapshot after it is downloaded.

I added a "customer-found" tag as this was identified via a ticket. I'd argue that this really shouldn't be in a "won't fix" status though.

The reason someone would specify an alternate location for a snapshot dump is due to space constraints. If all the --dir parameter does is copy the dump after it's completed, it doesn't seem terribly helpful. If anything, the help documentation should indicate what's actually happening - a copy to a folder, not download to folder:

[root@fuel ~]# fuel snapshot help
usage: fuel snapshot [-h] [--dir DIR]

optional arguments:
  -h, --help show this help message and exit
  --dir DIR Directory to which download snapshot.

Examples:

 To download diagnostic snapshot:
            fuel snapshot

        To download diagnostic snapshot to specific directory:
            fuel snapshot --dir path/to/directory

Silently storing old dumps deep in the filesystem is also problematic, since nobody expects there to be multiple copies of snapshots floating around. Due to the /var filesystem issue that many people are encountering, it's even more important to be mindful to clean up scratch space periodically and programmatically. Or at the very least, fully document the behavior, so that people know they have to manually clean up Fuel's snapshot scratch space.

+1 to comment #3, reopened.

Let me explain what is going on:

1. Snapshot is served by nginx server, that is why it is stored in /var/www/nailgun
2. Fuel client downloads that snapshot from nginx - it is not copied locally or anything
3. I hope it is clear that we can not know what user actually wants - download snapshot into specific directory
on local machine or put snapshot in another directory on master

So the only option that i see is to provide additional flag, like local, or overwrite snapshot settings.
But in such case snapshot wont be downloadable.

Hi Dima,

I understand that because of the current fuel client design it can't avoid
creating a snapshot in the default location. The client just goes to nginx and
downloads the snapshot to the specified directory.

What we actually ask here for is to change this design. We want to have a command line tool.
Preferably it should be the same "fuel snapshot" command for uniformity.
We want this command not to produce files anywhere except the directory we specified in the command line.
We need it for the cases when fuel ran out of space on /var/log, but we can attach another storage
to the fuel node and store the snapshot there.

Please seriously consider implementation of this feature.

Thanks,
Victor.

Guys fuel snapshot will generate it in mcollective container, and as you may know it doesnt have all hosts volumes mounted..

In my opinion we have better chance of covering such issues with shotgun directly (it is actually what performs logs collection and archiving).

Right now it is not installed on master host, but it could be with
>> yum install shotgun
or we even can preinstall during bootstrap

And config for shotgun is hidden in nailgun, but we can expose it with api call.

Just to summarize, we need:

1. Install shotgun on master host
2. Expose shotgun config with nailgun api call, like /api/logs/config/default
3. Add documentation on how to use it and change directory where logs are collected

Related fix proposed to branch: master
Review: https://review.openstack.org/157324

Assigning back to fuel-python, will get back to them ater ff

It looks like exposure of shotgun is the best we can do. It's a very useful tool. Will we have a console command for dumping the default config?

FUEL team, we found that shotgun is powerful tool, that we can use for generating custom diagnostic snapshots, with data we need.
To do this, please add ability to save config file on disk(I mean /tmp/dump_config that is created when snapshot task is creating ). File should be formatted to be human-readble, e.g. like with 'python -mjson.tool'
We can then modify config, including path to store snapshot, and run
shotgun -c /path/to/file
This is low-hanging fruit, please prioritize this.

Yes guys, thats the plan

Related fix proposed to branch: stable/6.0
Review: https://review.openstack.org/162930

Reviewed: https://review.openstack.org/157324
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=b81a7ca70386b09f35e4aab943018e98d3bd29fc
Submitter: Jenkins
Branch: master

commit b81a7ca70386b09f35e4aab943018e98d3bd29fc
Author: Dmitry Shulyak <email address hidden>
Date: Thu Feb 19 12:46:07 2015 +0200

    Expose default snapshot config and allow to pass to snapshot request

    Config will be exposed and dumped into file by fuel client.
    After this is done, it will be directly consumed by shotgun
    or some storage right in nailgun will be implemented to use
    user defined config

    Change-Id: I5e01a7459cefe49a128192d82dd827f02866f909
    Related-Bug: 1382511
    Related-Bug: 1420054

Related fix proposed to branch: stable/6.0
Review: https://review.openstack.org/162938

Reviewed: https://review.openstack.org/162930
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=bd69e7dda1e14a84f120e57bd6fd8678acc994ec
Submitter: Jenkins
Branch: stable/6.0

commit bd69e7dda1e14a84f120e57bd6fd8678acc994ec
Author: Dmitry Shulyak <email address hidden>
Date: Thu Feb 19 12:46:07 2015 +0200

    Expose default snapshot config and allow to pass to snapshot request

    Config will be exposed and dumped into file by fuel client.
    After this is done, it will be directly consumed by shotgun
    or some storage right in nailgun will be implemented to use
    user defined config

    Change-Id: I5e01a7459cefe49a128192d82dd827f02866f909
    Related-Bug: 1382511
    Related-Bug: 1420054

Fix proposed to branch: master
Review: https://review.openstack.org/162951

Related fix proposed to branch: master
Review: https://review.openstack.org/162954

Related fix proposed to branch: master
Review: https://review.openstack.org/162959

Change abandoned by Dmitry Shulyak (<email address hidden>) on branch: master
Review: https://review.openstack.org/162954

Change abandoned by Dmitry Shulyak (<email address hidden>) on branch: master
Review: https://review.openstack.org/162959

Related fix proposed to branch: stable/6.0
Review: https://review.openstack.org/163435

Reviewed: https://review.openstack.org/157738
Committed: https://git.openstack.org/cgit/stackforge/python-fuelclient/commit/?id=bb5434f0cfefd41ee53518f936332ed2d74b1abe
Submitter: Jenkins
Branch: master

commit bb5434f0cfefd41ee53518f936332ed2d74b1abe
Author: Dmitry Shulyak <email address hidden>
Date: Fri Feb 20 14:08:51 2015 +0200

    Add commands to operate snapshot config

    Added two commands to dump snapshot config
    and provide it for snapshot task

    Will download default config and dump it to stdout

    fuel snapshot --conf > conf.yaml

    Next command will accept config as input and bypass it to snapshot
    generation request

    fuel snapshot < conf.yaml

    This is required to overwrite certain parameters if needed,
    some of those are:
    - target directory of snapshot generation
    - timeout of snapshot generation procedure
    - exclusion/inclusion of certain directories

    Depends on I5e01a7459cefe49a128192d82dd827f02866f909
    Related-Bug: 1382511
    Related-Bug: 1420054
    DocImpact

    Change-Id: I1ddab26fba1346dad30955289e7d28f4d3aa1562

Change abandoned by Dmitry Shulyak (<email address hidden>) on branch: stable/6.0
Review: https://review.openstack.org/162938

Reviewed: https://review.openstack.org/163435
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=f5f2e9080c51b9044983e63f7f32870e3b256a5b
Submitter: Jenkins
Branch: stable/6.0

commit f5f2e9080c51b9044983e63f7f32870e3b256a5b
Author: Dmitry Shulyak <email address hidden>
Date: Thu Feb 19 12:46:07 2015 +0200

    Expose default snapshot config and allow to pass to snapshot request

    Config will be exposed and dumped into file by fuel client.
    After this is done, it will be directly consumed by shotgun
    or some storage right in nailgun will be implemented to use
    user defined config

    Original-change: I5e01a7459cefe49a128192d82dd827f02866f909
    Cant assign same one, because it was reverted one time

    Change-Id: I44f6b426d67bda62d54cbf4a751a78b689e06642
    Related-Bug: 1382511
    Related-Bug: 1420054

Reviewed: https://review.openstack.org/162938
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=f48ffff2efd0436146942ec33049b23783a390e6
Submitter: Jenkins
Branch: stable/6.0

commit f48ffff2efd0436146942ec33049b23783a390e6
Author: Dmitry Shulyak <email address hidden>
Date: Fri Feb 20 14:08:51 2015 +0200

    Add commands to operate snapshot config

    Added two commands to dump snapshot config
    and provide it for snapshot task

    Will download default config and dump it to stdout

    fuel snapshot --conf > conf.yaml

    Next command will accept config as input and bypass it to snapshot
    generation request

    fuel snapshot < conf.yaml

    This is required to overwrite certain parameters if needed,
    some of those are:
    - target directory of snapshot generation
    - timeout of snapshot generation procedure
    - exclusion/inclusion of certain directories

    Related-Bug: 1382511
    Related-Bug: 1420054

    Change-Id: I1ddab26fba1346dad30955289e7d28f4d3aa1562

Dmitry, I checked and customized snapshot works.
You rock!
Could you please add this important change to 5.1.2 and 5.0.3 branches too?

Dmitriy, one more request:
can we add config file that was used by snpahot to the snapshot?
Since snapshots can be not standard now, we need to know what config was run.

Ok, since fuel components do no updating when installing upgrade tar it doesn't make sense to include commit to 5.1.2 and 5.0.3.

Hi, Dmitriy.
Is fuel snapshot able to write somwhere, outside /var/www/nailgun/dump?
I tried to do snapshot with this target in config:
cat /tmp/superconfig | grep target
target: /tmp/fuel-snapshot

And got the following error:
[root@fuel ~]# fuel snapshot < /tmp/superconfig
Generating dump...
Traceback (most recent call last):
  File "/usr/bin/fuel", line 9, in <module>
    load_entry_point('fuelclient==6.0.0', 'console_scripts', 'fuel')()
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 93, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 218, in main
    parser.parse()
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/parser.py", line 102, in parse
    actions[parsed_params.action].action_func(parsed_params)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/base.py", line 59, in action_func
    method(params)
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/actions/snapshot.py", line 66, in get_snapshot
    directory=params.dir
  File "/usr/lib/python2.6/site-packages/fuelclient/cli/formatting.py", line 116, in download_snapshot_with_progress_bar
    download_handle = urllib2.urlopen(url)
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

And inside /var/log/nginx/error_nailgun.log
I see this error:
2015/03/22 03:50:40 [error] 350#0: *1140 open() "/var/www/nailgun/dump/fuel-snapshot-2015-03-22_03-48-53.tgz" failed (2: No such file or directory), client: 172.17.42.1, server: localhost, request: "GET /dump/fuel-snapshot-2015-03-22_03-48-53.tgz HTTP/1.1", host: "10.20.7.2:8000"
2015/03/22 03:51:17 [info] 350#0: *1141 client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 172.17.42.1, server: localhost, request: "PUT /api/nodes/agent/ HTTP/1.1", upstream: "uwsgi://10.20.7.2:8001", host: "10.20.7.2:8000"
2015/03/22 03:51:24 [info] 350#0: *1143 client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 172.17.42.1, server: localhost, request: "PUT /api/nodes/agent/ HTTP/1.1", upstream: "uwsgi://10.20.7.2:8001", host: "10.20.7.2:8000"

Also, fuel snapshot trying to download snapshot to current dir, so we still have 2 copies:
one is where target config parameter specified
second will be copied to current dir.

Related fix proposed to branch: master
Review: https://review.openstack.org/168262

There is only a several volumes that can be used inside of mcollective container (which is performs dumping).
And even more - only /var/www is served by nginx - any other directory will probably result in not found message.

If you want to store information in arbitrary place - you will need to use shotgun directly.

  yum install -y shotgun
  shotgun -c shotgun.json

And ofcource there should be a way to generate json config from fuel.

Change abandoned by Dmitry Pyzhov (<email address hidden>) on branch: master
Review: https://review.openstack.org/168262
Reason: No activity for more than a month

Reviewed: https://review.openstack.org/162951
Committed: https://git.openstack.org/cgit/openstack/fuel-docs/commit/?id=0b19b1516500f1fc803253073455f56b6071a4df
Submitter: Jenkins
Branch: master

commit 0b19b1516500f1fc803253073455f56b6071a4df
Author: Dmitry Shulyak <email address hidden>
Date: Tue Mar 10 12:45:41 2015 +0200

    Add basic doc on using shotgun

    Using shotgun itself solves several problems that can not
    be solved by using fuel api

    Change-Id: Idd9a0292b46a1890eeba03692c8d8b401d3ab947
    Closes-Bug: 1420054