Command execution timeout is hardcoded in shotgun, need to be able to override it in the config

Alex,
will you get a chance to make this happen any time soon? We are about to declare SCF, and though I would very much support this fix to go, the team will be fully focused on high/critical bugs.

Fix proposed to branch: master
Review: https://review.openstack.org/226270

Reviewed: https://review.openstack.org/226270
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=8df6aacaf51cca5e92d03a12b8ba47b29a19d2f9
Submitter: Jenkins
Branch: master

commit 8df6aacaf51cca5e92d03a12b8ba47b29a19d2f9
Author: Sebastian Kalinowski <email address hidden>
Date: Tue Sep 22 13:43:49 2015 +0200

    Allow to specify timeout for commands in shotgun

    Added new global setting DEFAULT_TIMEOUT for setting
    default timeout (10s) for executing command.

    It's also possible to specify timeout for all commands in
    snapshot by using 'timeout' option in snapshot config file.

    Additionally it's also possible to set command-specific timeout
    by using 'timeout' option in command config.

    Change-Id: If896485ea3b20ddde7bfc955a0a12b1faef98ea8
    Closes-Bug: #1441954

I think we should re-open this bug, because all of the tests bellow failed.

Case 1:
1) Go to the nailgun container on the master node
2) Edit file /usr/lib/python2.6/site-packages/nailgun/settings.yaml:
    Add this code to the DUMP section to the objects:
      - type: command
        command: sleep 10
        to_file: sleep.txt
        timeout: 5

   Save the file
3) Restart nailgun inside container: supervisorctl restart nailgun
4) SSH to the master node and execute: fuel snapshot

Expected result:
Fuel snapshot should fail because of an operation timeout
Actual result:
There are some errors at the shotgan logs inside mcollective container:
2015-11-03 09:50:25 DEBUG 18297 (driver) Running remote command: host: 10.109.0.2 command: sleep 10
2015-11-03 09:50:30 ERROR 18297 (driver) Error occured: Command failed to finish in 5 seconds

Case 2:
1) Go to the nailgun container on the master node
2) Edit file /usr/lib/python2.6/site-packages/nailgun/settings.yaml:
    Add this code to the DUMP section to the objects:
      - type: command
        command: sleep 11
        to_file: sleep.txt

   Save the file
3) Restart nailgun inside container: supervisorctl restart nailgun
4) Ensure that DEFAULT_TIMEOUT variable is set to 10 secs at the mcollective container /usr/lib/python2.6/site-packages/shotgun/settings.py file
5) SSH to the master node and execute: fuel snapshot

Expected result:
Fuel snapshot command should fail, at least there should be an error message at the logs like at case 1.
Actual result:
Fuel snapshot continues it's work. There are no errors at the shotgun log:
2015-11-03 09:57:33 DEBUG 20454 (driver) Running remote command: host: 10.109.0.2 command: sleep 11
2015-11-03 09:57:44 DEBUG 20454 (utils) Trying to execute command: mkdir -p "/var/www/nailgun/dump/fuel-snapshot-2015-11-03_09-57-30/10.109.0.2/commands"

Case 3:
The same as case 2, but set DEFAULT_TIMEOUT=0 at the mcollective container /usr/lib/python2.6/site-packages/shotgun/settings.py file

Actual and expected results are the same as for case 2. All commands ignore this default timeout.

Maybe I'm wrong about should `fuel snapshot` continue it's work after such errors, or not. But case 2 tells us about default timeout is not working.

Fuel used:
VERSION:
      api: '1.0'
      astute_sha: 0f753467a3f16e4d46e7e9f1979905fb178e4d5b
      build_id: '134'
      build_number: '134'
      feature_groups:
      - mirantis
      fuel-agent_sha: e881f0dabd09af4be4f3e22768b02fe76278e20e
      fuel-createmirror_sha: df6a93f7e2819d3dfa600052b0f901d9594eb0db
      fuel-library_sha: 14b6b2d55840f5271693e26a6544cc43c91989c4
      fuel-nailgun-agent_sha: d66f188a1832a9c23b04884a14ef00fc5605ec6d
      fuel-nailgun_sha: a0f7a4d5dfc8a78846fe57fa6245829643167505
      fuel-ostf_sha: 41aa5059243cbb25d7a80b97f8e1060a502b99dd
      fuelmain_sha: 01c8a575e342f867552075f136826e9b39cda631
      openstack_version: 2015.1.0-8.0
      production: docker
      python-fuelclient_sha: a8298df7de84960f32dccb1cd070255d5828a8ea
      release: '8.0'

We passed SCF in 8.0. Moving the bug to 9.0.