Please apply nofile limit fix of ceph-osd to trusty, utopic

Bug #1420647 reported by Nobuto Murata
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ceph (Ubuntu)
Fix Released
Medium
Dave Chiluk
Trusty
Fix Released
Medium
James Page
Utopic
Fix Released
Medium
James Page

Bug Description

[Impact]
Without this nofile limit fix, large ceph cluster fails to start or stops handling requests.

[Fix]
Ceph upstream applied a new nofile limit(10x) to Giant+ releases (vivid in Ubuntu repo).
https://github.com/ceph/ceph/pull/2156/files

[Test case]
Check that new nofile ulimit is properly applied as 32768 -> 327680:
$ cat /proc/$(pgrep -n ceph-osd)/limits | grep 'Max open files'

[Regression Potential]
minimal,
If we set 1M nofile ulimit, the daemon will fail to start due to the default fs.nr_open value. However this time, we will apply 0.3M which should not cause troubles.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: ceph 0.80.7-0ubuntu0.14.04.1
ProcVersionSignature: Ubuntu 3.13.0-44.73-generic 3.13.11-ckt12
Uname: Linux 3.13.0-44-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
Date: Wed Feb 11 07:14:43 2015
Ec2AMI: ami-00000016
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.smallswap
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
ExecutablePath: /usr/bin/ceph-osd
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
SourcePackage: ceph
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Nobuto Murata (nobuto) wrote :
Dave Chiluk (chiluk)
Changed in ceph (Ubuntu):
assignee: nobody → Dave Chiluk (chiluk)
tags: added: cts
Revision history for this message
Dave Chiluk (chiluk) wrote :
Revision history for this message
Dave Chiluk (chiluk) wrote :
Revision history for this message
Dave Chiluk (chiluk) wrote :

Above are the trusty and utopic debdiffs. The versioning is a bit different than I'm used to so, please feel free to correct the versioning.

Changed in ceph (Ubuntu):
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "trusty debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Robie Basak (racb) wrote :

Hi Dave,

Thank you for the debdiffs - they look good. A few questions:

Is this issue fixed in Vivid? I'm not sure if this is what you mean by "Ceph upstream applied a new nofile limit(10x) to Giant+ releases (vivid in Ubuntu repo)." - has this fix landed in Vivid, or is it just upstream and expected to land in Vivid?

dep3 header in "Origin: http..." should probably say "Origin: upstream, http..." to make it clear the patches are from upstream. Patches from upstream carry more confidence in cherry-picking to Ubuntu, so we should point this out for anybody trying to figure out the patches later. Don't worry about redoing the debdiffs just for this - I can just tweak before upload.

Changing the upstart script will presumably cause a conffile prompt if the user has made local modifications to the file. We should point this out in [Regression Potential] and make sure the SRU team know about it in order to help make a decision. We'd like to avoid conffile prompts in SRUs in general, but in this case it might make sense. I'll leave for the SRU team to decide.

The Trusty version should be 0.80.8-0ubuntu0.14.04.2 but you got the Utopic one right. I can tweak the Trusty version on upload. But there's already a ceph SRU in flight, so an upload will have to wait until that has gone through. I'll hold on reviewing the patch in detail and build testing until then.

Revision history for this message
Nobuto Murata (nobuto) wrote :

@Robie,
> Is this issue fixed in Vivid? I'm not sure if this is what you mean by "Ceph upstream applied a new nofile limit(10x) to Giant+ releases (vivid in Ubuntu repo)." - has this fix landed in Vivid, or is it just upstream and expected to land in Vivid?

This fix had already landed into vivid as a part of a new upstream release at 0.87-0ubuntu1/vivid.
http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/ceph/vivid/revision/63#src/upstart/ceph-osd.conf

Changed in ceph (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
James Page (james-page) wrote :

Pulling this in with 0.80.9 point release

Changed in ceph (Ubuntu Trusty):
status: New → In Progress
Changed in ceph (Ubuntu Utopic):
status: New → In Progress
Changed in ceph (Ubuntu Trusty):
importance: Undecided → Medium
Changed in ceph (Ubuntu Utopic):
importance: Undecided → Medium
Changed in ceph (Ubuntu Trusty):
assignee: nobody → James Page (james-page)
Changed in ceph (Ubuntu Utopic):
assignee: nobody → James Page (james-page)
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Nobuto, or anyone else affected,

Accepted ceph into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/0.80.9-0ubuntu0.14.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ceph (Ubuntu Utopic):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Nobuto, or anyone else affected,

Accepted ceph into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/0.80.9-0ubuntu0.14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ceph (Ubuntu Trusty):
status: In Progress → Fix Committed
Dave Chiluk (chiluk)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 0.80.9-0ubuntu0.14.04.1

---------------
ceph (0.80.9-0ubuntu0.14.04.1) trusty; urgency=medium

  [ James Page ]
  * New upstream stable point release (LP: #1413917).

  [ Dave Chiluk ]
  * Increase file limit to prevent hitting the file limit on large
    installations (LP: #1420647)
 -- James Page <email address hidden> Wed, 11 Mar 2015 09:14:35 +0000

Changed in ceph (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 0.80.9-0ubuntu0.14.10.1

---------------
ceph (0.80.9-0ubuntu0.14.10.1) utopic; urgency=medium

  [ James Page ]
  * New upstream stable point release (LP: #1413917).

  [ Dave Chiluk ]
  * Increase file limit to prevent hitting the file limit on large
    installations (LP: #1420647)
 -- James Page <email address hidden> Wed, 11 Mar 2015 09:16:00 +0000

Changed in ceph (Ubuntu Utopic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.