Add backport for (s390x) endian fixes to ceph nautilus

Bug #1851290 reported by Frank Heimes
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Medium
Unassigned
Train
Fix Released
Medium
Unassigned
Ussuri
Fix Released
Medium
Unassigned
Ubuntu on IBM z Systems
Fix Released
Undecided
Unassigned
ceph (Ubuntu)
Fix Released
Medium
Unassigned
Eoan
Fix Released
Medium
Unassigned

Bug Description

[Impact]
Ceph is not functional on big-endian deployments (s390x)

[Test Case]
Deploy ceph on a s390x (using Juju charms).
Monitors will fail to start, OSD's won't bootstrap.

[Regression Potential]
Low, patch has had lots of review upstream and has been accepted into the nautilus stable branch which is the release Ubuntu has in Focal and Eoan.

[Original Bug Report]
Since 'luminous' ceph (and in particular ceph-mon) fails on the s390x big endian platform.

A bug for that was already raised in 2017 - LP 1713032:
"[luminous] ceph-mon crashes when it is elected leader (s390x)"
https://bugs.launchpad.net/cloud-archive/+bug/1713032

It was a longer running task to get things fixed for s390x.
And finally uweigand (many thx Uli) got everything accepted upstream:
"Fix big-endian handling #30079" - https://github.com/ceph/ceph/pull/30079

In parallel to that there is also a backport of these s390x endianess fixes for ceph 'nautilus':
"nautilus: backport endian fixes #30697" - https://github.com/ceph/ceph/pull/30697

This ticket is to get this upstream checked, tested and partly approved 'nautilus' backport integrated into the UCA ceph 'nautilus'.

CVE References

Frank Heimes (fheimes)
summary: - Add backport for (s390x) endian fixes to nautilus
+ Add backport for (s390x) endian fixes to ceph nautilus
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ceph (Ubuntu):
status: New → Confirmed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 14.2.4-0ubuntu2

---------------
ceph (14.2.4-0ubuntu2) focal; urgency=medium

  [ Dariusz Gadomski ]
  * d/p/issue37490.patch: Cherry pick fix to optimize LVM queries in
    ceph-volume, resolving performance issues in systems under heavy load
    or with large numbers of disks (LP: #1850754).

  [ James Page ]
  * d/p/issue40114.patch: Cherry pick endian fixes to resolve issues
    using Ceph on big-endian architectures such as s390x (LP: #1851290).

 -- Dariusz Gadomski <email address hidden> Wed, 06 Nov 2019 11:08:53 +0100

Changed in ceph (Ubuntu):
status: Confirmed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → Fix Released
James Page (james-page)
description: updated
Changed in ceph (Ubuntu Eoan):
status: New → Triaged
importance: Undecided → High
Changed in ceph (Ubuntu):
importance: Undecided → High
importance: High → Medium
Changed in ceph (Ubuntu Eoan):
importance: High → Medium
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Frank, or anyone else affected,

Accepted ceph into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/14.2.4-0ubuntu0.19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ceph (Ubuntu Eoan):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-eoan
bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-182584 severity-high targetmilestone-inin1910
removed: verification-needed verification-needed-eoan
Revision history for this message
Frank Heimes (fheimes) wrote :

The bugzilla-bridge accidentally removed the verification tags - now added them again.

tags: added: verification-needed verification-needed-eoan
James Page (james-page)
Changed in cloud-archive:
status: Fix Released → Fix Committed
Revision history for this message
Andrew McLeod (admcleod) wrote :

I have tested this with Eoan proposed - ceph-mon now bootstraps and ceph reports that an OK health status:

https://pastebin.ubuntu.com/p/9brKDGmHcf/

ceph-osd also passes a storage test once related to this particular ceph-mon, when the following was executed from one of the ceph-mon nodes:

$ sudo ceph osd pool create rbd 128
$ echo 123456789 > /tmp/input.txt
$ sudo rados put -p rbd test_input /tmp/input.txt
$ sudo rados get -p rbd test_input /dev/stdout
123456789

Revision history for this message
Frank Heimes (fheimes) wrote :

Many thx Andrew - looks good.
I'm adjusting the tags accordingly.

tags: added: verification-done verification-done-eoan
removed: verification-needed verification-needed-eoan
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Frank, or anyone else affected,

Accepted ceph into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-train-needed
bugproxy (bugproxy)
tags: removed: verification-train-needed
Revision history for this message
Frank Heimes (fheimes) wrote :

The bugzilla-bridge again accidentally removed a verification tag - added it again.
(Problem is already reported.)

tags: added: verification-train-needed
bugproxy (bugproxy)
tags: added: verification-needed-train
removed: verification-train-needed
Frank Heimes (fheimes)
tags: added: verification-train-needed
removed: verification-needed-train
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 14.2.4-0ubuntu0.19.10.1

---------------
ceph (14.2.4-0ubuntu0.19.10.1) eoan; urgency=medium

  [ Dariusz Gadomski ]
  * d/p/issue37490.patch: Cherry pick fix to optimize LVM queries in
    ceph-volume, resolving performance issues in systems under heavy load
    or with large numbers of disks (LP: #1850754).

  [ James Page ]
  * New upstream release (LP: #1850901):
    - d/p/CVE-2019-10222.patch,ceph-volume-wait-for-lvs.patch,
      more-py3-compat.patch: Drop, included upstream.
    - d/p/bluefs-use-uint64_t-for-len.patch: Cherry pick fix to resolve
      FTBFS on 32 bit architectures.
  * d/p/issue40114.patch: Cherry pick endian fixes to resolve issues
    using Ceph on big-endian architectures such as s390x (LP: #1851290).

 -- James Page <email address hidden> Fri, 22 Nov 2019 09:59:04 +0000

Changed in ceph (Ubuntu Eoan):
status: Fix Committed → Fix Released
bugproxy (bugproxy)
tags: added: verification-needed-train
removed: verification-train-needed
Revision history for this message
Andrew McLeod (admcleod) wrote :

Verified functionality in bionic, train-proposed, both status and a functional test (storage) work OK -

Same test and results in comment #5, Test Case is OK

tags: added: verification-done-train
removed: verification-needed-train
Revision history for this message
James Page (james-page) wrote :

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
James Page (james-page) wrote :

This bug was fixed in the package ceph - 14.2.4-0ubuntu0.19.10.1~cloud0
---------------

 ceph (14.2.4-0ubuntu0.19.10.1~cloud0) bionic-train; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (14.2.4-0ubuntu0.19.10.1) eoan; urgency=medium
 .
   [ Dariusz Gadomski ]
   * d/p/issue37490.patch: Cherry pick fix to optimize LVM queries in
     ceph-volume, resolving performance issues in systems under heavy load
     or with large numbers of disks (LP: #1850754).
 .
   [ James Page ]
   * New upstream release (LP: #1850901):
     - d/p/CVE-2019-10222.patch,ceph-volume-wait-for-lvs.patch,
       more-py3-compat.patch: Drop, included upstream.
     - d/p/bluefs-use-uint64_t-for-len.patch: Cherry pick fix to resolve
       FTBFS on 32 bit architectures.
   * d/p/issue40114.patch: Cherry pick endian fixes to resolve issues
     using Ceph on big-endian architectures such as s390x (LP: #1851290).

Revision history for this message
James Page (james-page) wrote :

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
James Page (james-page) wrote :

This bug was fixed in the package ceph - 14.2.4-0ubuntu0.19.10.1~cloud0
---------------

 ceph (14.2.4-0ubuntu0.19.10.1~cloud0) bionic-train; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (14.2.4-0ubuntu0.19.10.1) eoan; urgency=medium
 .
   [ Dariusz Gadomski ]
   * d/p/issue37490.patch: Cherry pick fix to optimize LVM queries in
     ceph-volume, resolving performance issues in systems under heavy load
     or with large numbers of disks (LP: #1850754).
 .
   [ James Page ]
   * New upstream release (LP: #1850901):
     - d/p/CVE-2019-10222.patch,ceph-volume-wait-for-lvs.patch,
       more-py3-compat.patch: Drop, included upstream.
     - d/p/bluefs-use-uint64_t-for-len.patch: Cherry pick fix to resolve
       FTBFS on 32 bit architectures.
   * d/p/issue40114.patch: Cherry pick endian fixes to resolve issues
     using Ceph on big-endian architectures such as s390x (LP: #1851290).

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-02-06 10:57 EDT-------
IBM Bugzilla status ->closed, Fix Released with Nautilus

Changed in cloud-archive:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.