PostgreSQL Charm

PostgreSQL config from previous install found at /etc/postgresql/10/main/postgresql.conf

Bug #1831858 reported by Tytus Kurek on 2019-06-06

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	PostgreSQL Charm	Triaged	Medium	Unassigned

Bug Description

I've just found the following problem when deploying postgresql charm with point in time recovery options:

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
postgresql maas-region1 maas-region1 2.5.1 unsupported 09:59:49+02:00

App Version Status Scale Charm Store Rev OS Notes
keystone 13.0.2 active 1 keystone jujucharms 301 ubuntu
mysql 5.7.20-29.24 active 1 percona-cluster jujucharms 276 ubuntu
postgresql 10.8 blocked 1 postgresql jujucharms 199 ubuntu
swift-proxy 2.17.1 active 1 swift-proxy jujucharms 83 ubuntu
swift-storage 2.17.1 active 1 swift-storage jujucharms 256 ubuntu

Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 0/lxd/0 172.18.0.109 5000/tcp Unit is ready
mysql/0* active idle 0/lxd/1 172.18.0.102 3306/tcp Unit is ready
postgresql/0* blocked idle 0/lxd/4 172.18.0.107 PostgreSQL config from previous install found at /etc/postgresql/10/main/postgresql.conf
swift-proxy/0* active idle 0/lxd/2 172.18.0.110 8080/tcp Unit is ready
swift-storage/0* active idle 0 172.18.0.108 Unit is ready

Machine State DNS Inst id Series AZ Message
0 started 172.18.0.108 ehptre bionic default Deployed
0/lxd/0 started 172.18.0.109 juju-bd1991-0-lxd-0 bionic default Container started
0/lxd/1 started 172.18.0.102 juju-bd1991-0-lxd-1 bionic default Container started
0/lxd/2 started 172.18.0.110 juju-bd1991-0-lxd-2 bionic default Container started
0/lxd/4 started 172.18.0.107 juju-bd1991-0-lxd-4 bionic default Container started

$ juju run --unit postgresql/0 "charms.reactive -p get_flags"
['apt.installed.python3-psycopg2',
'apt.installed.run-one',
'leadership.is_leader',
'leadership.set.coordinator',
'leadership.set.master',
'leadership.set.replication_password',
'postgresql.client.passwords_set',
'postgresql.cluster.kernel_settings.set',
'postgresql.cluster.locale.set',
'postgresql.replication.cloned',
'postgresql.replication.has_master',
'postgresql.replication.is_master',
'postgresql.replication.is_primary',
'postgresql.upgrade.systemd',
'postgresql.wal_e.enabled',
'postgresql.wal_e.swift',
'snap.installed.core',
'snap.installed.wal-e',
'snap.refresh.set',
'workloadstatus.blocked']

In order to reproduce the issue please follow the instructions below:

$ cat swift-minimal-with-keystone.yaml
series: bionic
machines:
  "0":
    constraints: tags=swift
    series: bionic
services:
  keystone:
    charm: cs:keystone
    num_units: 1
    options:
      admin-password: admin
      token-provider: fernet
      worker-multiplier: 0.25
    to:
    - lxd:0
  mysql:
    charm: cs:percona-cluster
    num_units: 1
    options:
      innodb-buffer-pool-size: 256M
      max-connections: 1000
    to:
    - lxd:0
  swift-proxy:
    charm: cs:swift-proxy
    num_units: 1
    options:
      replicas: 1
      zone-assignment: manual
    to:
    - lxd:0
  swift-storage:
    charm: cs:swift-storage
    num_units: 1
    options:
      block-device: sdb sdc sdd
      zone: 1
    to:
    - 0
relations:
  - [ "keystone:shared-db", "mysql:shared-db" ]
  - [ "keystone:identity-service", "swift-proxy:identity-service" ]
  - [ "swift-proxy:swift-storage", "swift-storage:swift-storage" ]

$ juju deploy swift-minimal-with-keystone.yaml
Resolving charm: cs:keystone
Resolving charm: cs:percona-cluster
Resolving charm: cs:swift-proxy
Resolving charm: cs:swift-storage
Executing changes:
- upload charm cs:keystone-301 for series bionic
- deploy application keystone on bionic using cs:keystone-301
- upload charm cs:percona-cluster-276 for series bionic
- deploy application mysql on bionic using cs:percona-cluster-276
- upload charm cs:swift-proxy-83 for series bionic
- deploy application swift-proxy on bionic using cs:swift-proxy-83
- upload charm cs:swift-storage-256 for series bionic
- deploy application swift-storage on bionic using cs:swift-storage-256
- add new machine 0
- add relation keystone:shared-db - mysql:shared-db
- add relation keystone:identity-service - swift-proxy:identity-service
- add relation swift-proxy:swift-storage - swift-storage:swift-storage
- add unit swift-storage/0 to new machine 0
- add lxd container 0/lxd/0 on new machine 0
- add lxd container 0/lxd/1 on new machine 0
- add lxd container 0/lxd/2 on new machine 0
- add unit keystone/0 to 0/lxd/0
- add unit mysql/0 to 0/lxd/1
- add unit swift-proxy/0 to 0/lxd/2
Deploy of bundle completed.

$ cat postgresql.yaml
postgresql:
  wal_e_storage_uri: swift://backup
  os_username: admin
  os_password: admin
  os_auth_url: http://172.18.0.109:35357/v3
  os_tenant_name: admin

$ juju deploy postgresql --config postgresql.yaml
Located charm "cs:postgresql-199".
Deploying charm "cs:postgresql-199".

Attached is the log from the postgresql/0 unit.

Tags:

Revision history for this message

Tytus Kurek (tkurek) wrote on 2019-06-06:

unit-postgresql-0.log Edit (258.0 KiB, text/plain)

Revision history for this message

Tytus Kurek (tkurek) wrote on 2019-06-06:

Download full text (3.9 KiB)

Running the following command:

$ juju run --unit postgresql/0 "charms.reactive -p set_flag postgresql.cluster.initial-check"

moves the charm forward, however, it fails with the following errors:

2019-06-06 08:13:35 DEBUG update-status Authorization Failure. Authorization Failed: (http://172.18.0.109:35357/v3/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-b6f8785c-890a-400b-a8cc-568a96cad6f1)
2019-06-06 08:13:35 ERROR juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-1/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-postgresql-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-postgresql-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-postgresql-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-postgresql-1/charm/reactive/postgresql/wal_e.py", line 154, in ensure_swift_container
    subprocess.check_call(cmd, universal_newlines=True)
  File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/snap/bin/wal-e.envdir', '/etc/postgresql/10/main/wal-e.env', '/snap/bin/wal-e.swift', 'post', 'backup']' returned non-zero exit status 1.

Running the following command:

$ juju run --unit postgresql/0 "charms.reactive -p set_flag postgresql.cluster.initial-check"

moves the charm forward, however, it fails with the following errors:

2019-06-06 08:13:35 DEBUG update-status /var/lib/juju/agents/unit-postgresql-1/charm/reactive/postgresql/helpers.py:107: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
2019-06-06 08:13:35 DEBUG update-status   return yaml.load(f)
2019-06-06 08:13:35 DEBUG update-status Traceback (most recent call last):
2019-06-06 08:13:35 DEBUG update-status   File "/var/lib/juju/agents/unit-postgresql-1/charm/hooks/update-status", line 22, in <module>
2019-06-06 08:13:35 DEBUG update-status     main()
2019-06-06 08:13:35 DEBUG update-status   File "/var/lib/juju/agents/unit-postgresql-1/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
2019-06-06 08:13:35 DEBUG update-status     bus.dispatch(restricted=restricted_mode)
2019-06-06 08:13:35 DEBUG update-status   File "/var/lib/juju/agents/unit-postgresql-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
2019-06-06 08:13:35 DEBUG update-status     _invoke(other_handlers)
2019-06-06 08:13:35 DEBUG update-status   File "/var/lib/juju/agents/unit-postgresql-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
2019-06-06 08:13:35 DEBUG update-status     handler.invoke()
2019-06-06 08:13:35 DEBUG update-status   File "/var/lib/juju/agents/unit-postgresql-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
2019-06-06 08:13:35 DEBUG update-status     self._action(*args)
2019-06-06 08:13:35 DEBUG update-status   File "/var/lib/juju/agents/unit-postgresql-1/charm/reactive/postgresql/wal_e.py", line 154, in ensure_swift_container
2019-06-06 08:13:35 DEBUG update-status     subprocess.check_call(cmd, universal_newlines=True)
2019-06-06 08:13:35 DEBUG update-status   File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
2019-06-06 08:13:35 DEBUG update-status     raise CalledProcessError(retcode, cmd)
2019-06-06 08:13:35 DEBUG update-status subprocess.CalledProcessError: Command '['/snap/bin/wal-e.envdir', '/etc/postgresql/10/main/wal-e.env', '/snap/bin/wal-e.swift', 'post', 'backup']' returned non-zero exit status 1.
2019-06-06 08:13:35 ERROR juju.worker.uniter.operation runhook.go:132 hook "update-status" failed: exit status 1

which is most likely a different issue.

Revision history for this message

Tytus Kurek (tkurek) wrote on 2019-06-06:

So the authorization problem is a result of other bug (https://bugs.launchpad.net/postgresql-charm/+bug/1831865), but it looks like manually setting the 'cluster.initial-check' flag solves the problem.

Revision history for this message

Tytus Kurek (tkurek) wrote on 2019-06-06:

The issue seems to be present on bionic only.

Revision history for this message

Stuart Bishop (stub) wrote on 2019-06-06:

2019-06-06 07:47:00 DEBUG install The following NEW packages will be installed:
2019-06-06 07:47:00 DEBUG install libpq5 python3-psycopg2

What is missing in the above are the PostgreSQL packages. PostgreSQL 10 had already been installed on the machine before the charm was started, and the charm refuses to destroy the existing database (by design, to avoid potential data loss).

A Juju action to destroy and recreate the database would be one way to have a supported way of getting past this blocked state, and useful for some other use cases like test suites.