Race in Postgres on master node in 5.0

Bug #1401844 reported by Sergii Golovatiuk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Unassigned
5.0.x
In Progress
High
Matthew Mosesohn
5.1.x
Invalid
Undecided
Unassigned
6.0.x
Invalid
Undecided
Unassigned
6.1.x
Invalid
High
Unassigned

Bug Description

When master node is virtualized and host has pretty high load. I see the race condition as

Notice: /Stage[main]/Main/Postgresql::Db[ostf]/Postgresql::Database[ostf]/Exec[/usr/bin/createdb --template=template0 --encoding 'utf8' --locale=C 'ostf']/returns: createdb: could not connect to database template1: FATAL: the database system is starting up
Error: /usr/bin/createdb --template=template0 --encoding 'utf8' --locale=C 'ostf' returned 1 instead of one of [0]
Error: /Stage[main]/Main/Postgresql::Db[ostf]/Postgresql::Database[ostf]/Exec[/usr/bin/createdb --template=template0 --encoding 'utf8' --locale=C 'ostf']/returns: change from notrun to 0 failed: /usr/bin/createdb --template=template0 --encoding 'utf8' --locale=C 'ostf' returned 1 instead of one of [0]
Notice: /Stage[main]/Main/Postgresql::Db[ostf]/Postgresql::Database[ostf]/Postgresql::Psql[REVOKE CONNECT ON DATABASE ostf FROM public]/Exec[/bin/echo "REVOKE CONNECT ON DATABASE ostf FROM public" | /usr/bin/psql --no-password --tuples-only --quiet --dbname postgres |egrep -v -q '^$']: Dependency Exec[/usr/bin/createdb --template=template0 --encoding 'utf8' --locale=C 'ostf'] has failures: true

It means that Postgres started but DBs were not attached though puppet ran next resource though DBs were not ready.

the problem was fixed in 5.1.1 by updating Postgres manifests to latest though it's present in 5.0 (all branches) and 5.1 (only 5.1)

The actual fixes are:
https://review.openstack.org/#/c/131076/4/deployment/puppet/nailgun/examples/postgres-only.pp

Here is a root cause of issue
https://github.com/puppetlabs/puppetlabs-postgresql/pull/292/files

Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Matthew Mosesohn (raytrac3r)
milestone: none → 5.0.3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/141325

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/141345

Revision history for this message
Ryan Moe (rmoe) wrote :

Marking invalid for all but 5.0.3 as the description mentions that this only a problem on 5.0.

Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

This bug will still affect 5.0.3. I couldn't merge this because we don't have a working 5.0 CI.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/5.0)

Change abandoned by Matthew Mosesohn (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/141345

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Matthew Mosesohn (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/141325

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.