postgres nodes fail install, retry works
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu CI Engine |
New
|
Undecided
|
Unassigned |
Bug Description
I've seen this with both the ppa-assigner and ticket system django nodes. If I do just a "juju resolved --retry", it usually works again without further intervention. It acts as if there is a race condition which occasionally causes dpkg to fail.
[log]
2014-03-03 21:20:11 DEBUG juju.agent agent.go:253 Reading agent config, format: format 1.16
2014-03-03 21:20:11 INFO juju.jujud unit.go:75 unit agent unit-ts-postgres-0 start (1.17.4-
2014-03-03 21:20:11 INFO juju runner.go:262 worker: start "api"
2014-03-03 21:20:11 INFO juju apiclient.go:113 state/api: dialing "wss://
2014-03-03 21:20:11 INFO juju apiclient.go:123 state/api: connection established
2014-03-03 21:20:11 DEBUG juju.agent agent.go:394 writing configuration file
2014-03-03 21:20:11 INFO juju runner.go:262 worker: start "upgrader"
2014-03-03 21:20:11 INFO juju runner.go:262 worker: start "logger"
2014-03-03 21:20:11 DEBUG juju.worker.logger logger.go:35 initial log config: "<root>=DEBUG"
2014-03-03 21:20:11 INFO juju runner.go:262 worker: start "uniter"
2014-03-03 21:20:11 DEBUG juju.worker.logger logger.go:60 logger setup
2014-03-03 21:20:11 INFO juju runner.go:262 worker: start "rsyslog"
2014-03-03 21:20:11 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>
2014-03-03 21:20:26 INFO juju-log Running install hook
2014-03-03 21:20:27 INFO install Get:1 http://
2014-03-03 21:20:27 INFO install Get:2 http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Get:3 http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Get:4 http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Ign http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Get:5 http://
2014-03-03 21:20:27 INFO install Ign http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Get:6 http://
2014-03-03 21:20:27 INFO install Get:7 http://
2014-03-03 21:20:27 INFO install Get:8 http://
2014-03-03 21:20:27 INFO install Get:9 http://
2014-03-03 21:20:27 INFO install Get:10 http://
2014-03-03 21:20:27 INFO install Get:11 http://
2014-03-03 21:20:27 INFO install Get:12 http://
2014-03-03 21:20:27 INFO install Get:13 http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Get:14 http://
2014-03-03 21:20:27 INFO install Get:15 http://
2014-03-03 21:20:27 INFO install Get:16 http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:27 INFO install Hit http://
2014-03-03 21:20:28 INFO install Fetched 3728 kB in 1s (2563 kB/s)
2014-03-03 21:20:40 INFO install Reading package lists...
2014-03-03 21:20:40 INFO juju-log map version from distro release ...
2014-03-03 21:20:40 INFO juju-log version=9.1 from distro_
Reading package lists... Doneall
Building dependency tree l
Reading state information... Done
2014-03-03 21:20:41 INFO juju-log Installing ['python-psutil', 'postgresql-9.1', 'postgresql-
2014-03-03 21:20:41 INFO install Reading package lists...
2014-03-03 21:20:41 INFO install Building dependency tree...
2014-03-03 21:20:41 INFO install Reading state information...
2014-03-03 21:20:41 INFO install The following extra packages will be installed:
2014-03-03 21:20:41 INFO install libcrypt-
2014-03-03 21:20:41 INFO install postgresql-
2014-03-03 21:20:41 INFO install python-
2014-03-03 21:20:41 INFO install syslinux-common
2014-03-03 21:20:41 INFO install Suggested packages:
2014-03-03 21:20:41 INFO install uuid floppyd oidentd ident-server locales-all postgresql-doc-9.1
2014-03-03 21:20:41 INFO install libdbd-pg-perl python-
2014-03-03 21:20:41 INFO install python-
2014-03-03 21:20:41 INFO install python-psycopg2-doc openssl-blacklist
2014-03-03 21:20:41 INFO install The following NEW packages will be installed:
2014-03-03 21:20:41 INFO install libcrypt-
2014-03-03 21:20:41 INFO install postgresql-9.1 postgresql-
2014-03-03 21:20:41 INFO install postgresql-common postgresql-
2014-03-03 21:20:41 INFO install python-
2014-03-03 21:20:41 INFO install python-markupsafe python-psutil python-psycopg2 ssl-cert syslinux
2014-03-03 21:20:41 INFO install syslinux-common
2014-03-03 21:20:41 INFO install 0 upgraded, 20 newly installed, 0 to remove and 0 not upgraded.
2014-03-03 21:20:41 INFO install Need to get 8523 kB of archives.
2014-03-03 21:20:41 INFO install After this operation, 23.6 MB of additional disk space will be used.
2014-03-03 21:20:41 INFO install Get:1 http://
2014-03-03 21:20:41 INFO install Get:2 http://
2014-03-03 21:20:44 INFO install Get:3 http://
2014-03-03 21:20:44 INFO install Get:4 http://
2014-03-03 21:20:44 INFO install Get:5 http://
2014-03-03 21:20:44 INFO install Get:6 http://
2014-03-03 21:20:44 INFO install Get:7 http://
2014-03-03 21:20:44 INFO install Get:8 http://
2014-03-03 21:20:44 INFO install Get:9 http://
2014-03-03 21:20:44 INFO install Get:10 http://
2014-03-03 21:20:44 INFO install Get:11 http://
2014-03-03 21:20:44 INFO install Get:12 http://
2014-03-03 21:20:44 INFO install Get:13 http://
2014-03-03 21:20:44 INFO install Get:14 http://
2014-03-03 21:20:44 INFO install Get:15 http://
2014-03-03 21:20:44 INFO install Get:16 http://
2014-03-03 21:20:44 INFO install Get:17 http://
2014-03-03 21:20:44 INFO install Get:18 http://
2014-03-03 21:20:44 INFO install Get:19 http://
2014-03-03 21:20:44 INFO install Get:20 http://
2014-03-03 21:20:44 INFO install Preconfiguring packages ...
2014-03-03 21:20:45 INFO install Fetched 8523 kB in 0s (26.0 MB/s)
2014-03-03 21:20:52 INFO install Selecting previously unselected package libxslt1.1.
2014-03-03 21:20:52 INFO install (Reading database ... 48202 files and directories currently installed.)
2014-03-03 21:20:52 INFO install Unpacking libxslt1.1 (from .../libxslt1.
2014-03-03 21:20:55 INFO install Selecting previously unselected package libcrypt-
2014-03-03 21:20:55 INFO install Unpacking libcrypt-
2014-03-03 21:21:02 INFO install Selecting previously unselected package libossp-uuid16.
2014-03-03 21:21:02 INFO install Unpacking libossp-uuid16 (from .../libossp-
2014-03-03 21:21:06 INFO install Selecting previously unselected package libpq5.
2014-03-03 21:21:06 INFO install Unpacking libpq5 (from .../libpq5_
2014-03-03 21:21:09 INFO install Selecting previously unselected package mtools.
2014-03-03 21:21:10 INFO install Unpacking mtools (from .../mtools_
2014-03-03 21:21:19 INFO install Selecting previously unselected package postgresql-
2014-03-03 21:21:19 INFO install Unpacking postgresql-
2014-03-03 21:21:22 INFO install Selecting previously unselected package postgresql-
2014-03-03 21:21:22 INFO install Unpacking postgresql-
2014-03-03 21:21:31 INFO install Selecting previously unselected package ssl-cert.
2014-03-03 21:21:31 INFO install Unpacking ssl-cert (from .../ssl-
2014-03-03 21:21:40 INFO install Selecting previously unselected package postgresql-common.
2014-03-03 21:21:40 INFO install Unpacking postgresql-common (from .../postgresql-
2014-03-03 21:21:40 INFO install Adding 'diversion of /usr/bin/pg_config to /usr/bin/
2014-03-03 21:21:47 INFO install Selecting previously unselected package postgresql-9.1.
2014-03-03 21:21:47 INFO install Unpacking postgresql-9.1 (from .../postgresql-
2014-03-03 21:21:56 INFO install Selecting previously unselected package postgresql-
2014-03-03 21:21:56 INFO install Unpacking postgresql-
2014-03-03 21:22:06 INFO install Selecting previously unselected package postgresql-
2014-03-03 21:22:06 INFO install Unpacking postgresql-
2014-03-03 21:22:13 INFO install Selecting previously unselected package python-
2014-03-03 21:22:13 INFO install Unpacking python-
2014-03-03 21:22:17 INFO install Selecting previously unselected package python-
2014-03-03 21:22:17 INFO install Unpacking python-
2014-03-03 21:22:21 INFO install Selecting previously unselected package python-markupsafe.
2014-03-03 21:22:21 INFO install Unpacking python-markupsafe (from .../python-
2014-03-03 21:22:24 INFO install Selecting previously unselected package python-jinja2.
2014-03-03 21:22:24 INFO install Unpacking python-jinja2 (from .../python-
2014-03-03 21:22:30 INFO install Selecting previously unselected package python-psutil.
2014-03-03 21:22:30 INFO install Unpacking python-psutil (from .../python-
2014-03-03 21:22:34 INFO install Selecting previously unselected package python-psycopg2.
2014-03-03 21:22:34 INFO install Unpacking python-psycopg2 (from .../python-
2014-03-03 21:22:37 INFO install Selecting previously unselected package syslinux-common.
2014-03-03 21:22:37 INFO install Unpacking syslinux-common (from .../syslinux-
2014-03-03 21:22:50 INFO install Selecting previously unselected package syslinux.
2014-03-03 21:22:50 INFO install Unpacking syslinux (from .../syslinux_
2014-03-03 21:22:53 INFO install Processing triggers for man-db ...
2014-03-03 21:22:56 INFO install Processing triggers for install-info ...
2014-03-03 21:22:58 INFO install Processing triggers for ureadahead ...
2014-03-03 21:22:59 INFO install Setting up libxslt1.1 (1.1.26-8ubuntu1.3) ...
2014-03-03 21:23:00 INFO install Setting up libcrypt-
2014-03-03 21:23:01 INFO install Setting up libossp-uuid16 (1.6.2-1.1build3) ...
2014-03-03 21:23:02 INFO install Setting up libpq5 (9.1.12-
2014-03-03 21:23:03 INFO install Setting up mtools (4.0.12-
2014-03-03 21:23:04 INFO install Setting up postgresql-
2014-03-03 21:23:05 INFO install Setting up postgresql-
2014-03-03 21:23:07 INFO install update-
2014-03-03 21:23:08 INFO install Setting up ssl-cert (1.0.28ubuntu0.1) ...
2014-03-03 21:23:11 INFO install Setting up postgresql-common (129ubuntu1) ...
2014-03-03 21:23:17 INFO install Adding user postgres to group ssl-cert
2014-03-03 21:23:18 INFO install Building PostgreSQL dictionaries from installed myspell/hunspell packages...
2014-03-03 21:23:19 INFO install Setting up postgresql-9.1 (9.1.12-
2014-03-03 21:23:20 INFO install Creating new cluster (configuration: /etc/postgresql
2014-03-03 21:23:22 INFO install Moving configuration file /var/lib/
2014-03-03 21:23:22 INFO install Moving configuration file /var/lib/
2014-03-03 21:23:22 INFO install Moving configuration file /var/lib/
2014-03-03 21:23:36 INFO install Configuring postgresql.conf to use port 5432...
2014-03-03 21:23:36 INFO install update-
2014-03-03 21:23:36 INFO install * Starting PostgreSQL 9.1 database server
2014-03-03 21:24:07 INFO install * The PostgreSQL server failed to start. Please check the log output.
2014-03-03 21:24:07 INFO install ...fail!
2014-03-03 21:24:07 INFO install invoke-rc.d: initscript postgresql, action "start" failed.
2014-03-03 21:24:07 INFO install dpkg: error processing postgresql-9.1 (--configure):
2014-03-03 21:24:07 INFO install subprocess installed post-installation script returned error exit status 1
2014-03-03 21:24:07 INFO install dpkg: dependency problems prevent configuration of postgresql-
2014-03-03 21:24:07 INFO install postgresql-
2014-03-03 21:24:07 INFO install Package postgresql-9.1 is not configured yet.
2014-03-03 21:24:07 INFO install dpkg: error processing postgresql-
2014-03-03 21:24:07 INFO install dependency problems - leaving unconfigured
2014-03-03 21:24:07 INFO install dpkg: dependency problems prevent configuration of postgresql-
2014-03-03 21:24:07 INFO install postgresql-
2014-03-03 21:24:07 INFO install Package postgresql-9.1 is not configured yet.
2014-03-03 21:24:07 INFO install dpkg: error processing postgresql-
2014-03-03 21:24:07 INFO install dependency problems - leaving unconfigured
2014-03-03 21:24:07 INFO install Setting up python-
2014-03-03 21:24:07 INFO install No apport report written because the error message indicates its a followup error from a previous failure.
2014-03-03 21:24:07 INFO install No apport report written because the error message indicates its a followup error from a previous failure.
2014-03-03 21:24:19 INFO install Setting up python-
2014-03-03 21:24:20 INFO install Setting up python-markupsafe (0.15-1) ...
2014-03-03 21:24:21 INFO install Setting up python-jinja2 (2.6-1) ...
2014-03-03 21:24:22 INFO install Setting up python-psutil (0.4.1-1ubuntu1) ...
2014-03-03 21:24:24 INFO install Setting up python-psycopg2 (2.4.5-1) ...
2014-03-03 21:24:25 INFO install Setting up syslinux-common (2:4.05+dfsg-2) ...
2014-03-03 21:24:25 INFO install Setting up syslinux (2:4.05+dfsg-2) ...
2014-03-03 21:24:27 INFO install Processing triggers for libc-bin ...
2014-03-03 21:24:27 INFO install ldconfig deferred processing now taking place
2014-03-03 21:24:28 INFO install Errors were encountered while processing:
2014-03-03 21:24:28 INFO install postgresql-9.1
2014-03-03 21:24:28 INFO install postgresql-
2014-03-03 21:24:28 INFO install postgresql-
2014-03-03 21:24:28 INFO install E: Sub-process /usr/bin/dpkg returned an error code (1)
2014-03-03 21:24:29 INFO install Traceback (most recent call last):
2014-03-03 21:24:29 INFO install File "/var/lib/
2014-03-03 21:24:29 INFO install hooks.execute(
2014-03-03 21:24:29 INFO install File "/var/lib/
2014-03-03 21:24:29 INFO install self._hooks[
2014-03-03 21:24:29 INFO install File "/var/lib/
2014-03-03 21:24:29 INFO install update_
2014-03-03 21:24:29 INFO install File "/var/lib/
2014-03-03 21:24:29 INFO install fetch.apt_
2014-03-03 21:24:29 INFO install File "/var/lib/
2014-03-03 21:24:29 INFO install subprocess.
2014-03-03 21:24:29 INFO install File "/usr/lib/
2014-03-03 21:24:29 INFO install raise CalledProcessEr
2014-03-03 21:24:29 INFO install subprocess.
2014-03-03 21:24:29 ERROR juju.worker.uniter uniter.go:480 hook failed: exit status 1
no longer affects: | ubuntu-ci-services-itself |
Also found that when this happens to the ts-postgres node, it can lead to failure to start the lander worker upstart job because the ts-django node isn't responding before the upstart job gives up.
May need to update the lander upstart job to never give up.