zookeeper connection is not using exponential backoff
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
txzookeeper |
New
|
Undecided
|
Unassigned | ||
juju (Ubuntu) |
Confirmed
|
Medium
|
Unassigned |
Bug Description
My juju cluster had some connection issues to zookeeper. While I was reading the charm.log of my jenkins-slave unit I noticed that juju had logged many thousands of exceptions such as this one:
2012-11-09 06:51:07,514: twisted@ERROR: Traceback (most recent call last):
2012-11-09 06:51:07,514: twisted@ERROR: File "/usr/lib/
2012-11-09 06:51:07,514: twisted@ERROR: if self._check_
2012-11-09 06:51:07,514: twisted@ERROR: File "/usr/lib/
2012-11-09 06:51:07,514: twisted@ERROR: self, error)
2012-11-09 06:51:07,515: twisted@ERROR: File "/usr/lib/
2012-11-09 06:51:07,515: twisted@ERROR: result = f(*args, **kw)
2012-11-09 06:51:07,515: twisted@ERROR: File "/usr/lib/
2012-11-09 06:51:07,515: twisted@ERROR: return _inlineCallback
2012-11-09 06:51:07,516: twisted@ERROR: --- <exception caught here> ---
2012-11-09 06:51:07,516: twisted@ERROR: File "/usr/lib/
2012-11-09 06:51:07,516: twisted@ERROR: result = g.send(result)
2012-11-09 06:51:07,516: twisted@ERROR: File "/usr/lib/
2012-11-09 06:51:07,517: twisted@ERROR: raise error
2012-11-09 06:51:07,517: twisted@ERROR: zookeeper.
I can see about 300 such exceptions _every second_. This is very bad on two levels:
1) It quickly fills the log with pointless exceptions, using disk space, saturating slow virtual IO
2) It is against proven network practice of using exponential backoff when retrying failed communication
ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: juju 0.5+bzr531-
ProcVersionSign
Uname: Linux 3.2.0-32-virtual x86_64
ApportVersion: 2.0.1-0ubuntu14
Architecture: amd64
Date: Tue Nov 13 09:26:31 2012
Ec2AMI: ami-000000bf
Ec2AMIManifest: FIXME
Ec2Availability
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitec
ProcEnviron:
TERM=xterm-
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: juju
UpgradeStatus: No upgrade log present (probably fresh install)
Changed in juju (Ubuntu): | |
importance: | Undecided → Medium |
Status changed to 'Confirmed' because the bug affects multiple users.