PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter

Bug #1355998 reported by Steve Martinelli
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Core Infrastructure
Fix Released
Critical
Jonathan Harker

Bug Description

All the logs: http://logs.openstack.org/53/113553/2/check/check-tempest-dsvm-postgres-full/271cc6b/

Full devstack log: http://logs.openstack.org/53/113553/2/check/check-tempest-dsvm-postgres-full/271cc6b/logs/devstacklog.txt.gz

snippet:
2014-08-12 17:16:59.790 | + restart_service postgresql
2014-08-12 17:16:59.790 | + is_ubuntu
2014-08-12 17:16:59.790 | + [[ -z deb ]]
2014-08-12 17:16:59.790 | + '[' deb = deb ']'
2014-08-12 17:16:59.790 | + sudo /usr/sbin/service postgresql restart
2014-08-12 17:16:59.800 | * Restarting PostgreSQL 9.1 database server
2014-08-12 17:17:01.371 | * The PostgreSQL server failed to start. Please check the log output:
2014-08-12 17:17:01.371 | 2014-08-12 17:17:01 UTC FATAL: could not create shared memory segment: Invalid argument
2014-08-12 17:17:01.371 | 2014-08-12 17:17:01 UTC DETAIL: Failed system call was shmget(key=5432001, size=36954112, 03600).
2014-08-12 17:17:01.371 | 2014-08-12 17:17:01 UTC HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 36954112 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2014-08-12 17:17:01.371 | If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
2014-08-12 17:17:01.371 | The PostgreSQL documentation contains more information about shared memory configuration.
2014-08-12 17:17:01.371 | ...fail!
2014-08-12 17:17:01.372 | + exit_trap

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/113634

Revision history for this message
Jeremy Stanley (fungi) wrote :

According to logstash this started just after the daily nodepool image updates completed, and is only happening on devstack-trusty nodes in hpcloud-b1. I have deleted that image and held one of the nodes booted from it (which reports 33554432 from /proc/sys/kernel/shmmax just like nodes booted from the previous day's image). I've gone over the image update log, which didn't contain any obvious configuration/setup errors, but will compare it to an earlier run and other hpcloud provider network versions of the same to see if there's a relevant difference.

Changed in openstack-ci:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Jeremy Stanley (fungi)
milestone: none → juno
Changed in openstack-ci:
assignee: Jeremy Stanley (fungi) → Jonathan Harker (jesusaurus)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.openstack.org/113678
Committed: https://git.openstack.org/cgit/openstack-infra/config/commit/?id=2fa821d2ce4b4f655a130668edefb3a663611202
Submitter: Jenkins
Branch: master

commit 2fa821d2ce4b4f655a130668edefb3a663611202
Author: K Jonathan Harker <email address hidden>
Date: Tue Aug 12 16:20:36 2014 -0700

    Accidentally turned a trusty image into precise

    Closes-Bug: #1355998
    Change-Id: Ic585555d48c374f0441615ba95f416ad5f7f8e37

Changed in openstack-ci:
status: In Progress → Fix Released
Jeremy Stanley (fungi)
no longer affects: devstack
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to elastic-recheck (master)

Reviewed: https://review.openstack.org/113634
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=d0cd62c9c3eaab7f9b1e2131f8d4f792cacf530b
Submitter: Jenkins
Branch: master

commit d0cd62c9c3eaab7f9b1e2131f8d4f792cacf530b
Author: Davanum Srinivas <email address hidden>
Date: Tue Aug 12 16:07:02 2014 -0400

    Add query for Postgres startup failure

    24 hits in last 24 hours

    Related-Bug: 1355998

    Change-Id: If4b364af11a85ec0ee94a7cae05f6c77fdde0722

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.