Installation should retry when failing to connect due to network transient issue

Bug #1453862 reported by Larry Michel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Charm Helpers
New
Undecided
Unassigned

Bug Description

Occasionally, there are network issues preventing connection to archives. The charm install will automatically fail. These network issues preventing contact with archive will sometimes persist for a short period of time and may be intermittent. If they persist beyond X minutes or so, then the charm should fail. But regardless of timeout, charm should retry.

Here's an example:

From console.txt:

2015-05-07 04:24:37,453 [WARNING] deployer.import: Config specifies num units for subordinate: neutron-calico
2015-05-07 04:24:37,454 [INFO] deployer.import: Adding 2 more units to swift-storage
2015-05-07 04:47:33,208 [ERROR] deployer.env: The following units had errors:
   unit: mysql/0: machine: 3/lxc/0 agent-state: error details: hook failed: "install"
2015-05-07 04:47:33,213 [ERROR] oil_ci.deploy.oil_deployer: Deployment failed:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oil_ci/deploy/oil_deployer.py", line 140, in deploy
    debug=True, verbose=True)
  File "/usr/lib/python2.7/dist-packages/oil_ci/juju/juju_deployer.py", line 85, in run_deployer
    importer.Importer(env, deploy, options=opts).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 206, in run
    self.wait_for_units(ignore_errors=ignore_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 181, in wait_for_units
    services=self.deployment.get_service_names(), on_errors=on_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 227, in wait_for_units
    services=services, on_errors=on_errors).run(callback)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 1923, in run
    if self.complete() is True:
  File "/usr/lib/python2.7/dist-packages/deployer/env/watchers.py", line 77, in complete
    on_errors(new_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/watchers.py", line 98, in callback
    raise ErrorExit()

From unit log:

unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 Err http://archive.ubuntu.com/ubuntu/ trusty/main python-jinja2 all 2.7.2-2
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 Connection failed [IP: 91.189.91.14 80]
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 Fetched 14.3 kB in 2min 13s (106 B/s)
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/j/jinja2/python-jinja2_2.7.2-2_all.deb Connection failed [IP: 91.189.91.14 80]
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 Traceback (most recent call last):
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/install", line 6, in <module>
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 import lib.utils as utils
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/utils.py", line 48, in <module>
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 install('python-jinja2')
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/utils.py", line 41, in install
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 subprocess.check_call(cmd)
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 raise CalledProcessError(retcode, cmd)
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 subprocess.CalledProcessError: Command '['apt-get', '-y', 'install', 'python-jinja2']' returned non-zero exit status 100
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO juju.worker.uniter.context context.go:381 handling reboot
unit-mysql-0[1981]: 2015-05-07 04:47:29 ERROR juju.worker.uniter.operation runhook.go:89 hook "install" failed: exit status 1
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO juju.worker.uniter modes.go:452 ModeContinue exiting
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO juju.worker.uniter modes.go:450 ModeHookError starting
unit-mysql-0[1981]: 2015-05-07 04:47:29 DEBUG juju.worker.uniter.filter filter.go:597 want resolved event
unit-mysql-0[1981]: 2015-05-07 04:47:29 DEBUG juju.worker.uniter.filter filter.go:591 want forced upgrade true
unit-mysql-0[1981]: 2015-05-07 04:47:29 DEBUG juju.worker.uniter.filter filter.go:727 no new charm event

Tags: oil
Revision history for this message
Michał Sawicz (saviq) wrote :

The same applies to the Git fetcher.

summary: - Installation should retry when failing to connect to archives due to
- network transient issue
+ Installation should retry when failing to connect due to network
+ transient issue
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.