Swift-storage dies if rsyslog is stopped

This appears to be a bug in the Swift logging behavior not in the charm itself.

There's an upstream commit located at [0] which fixes the problem. In a nutshell, the stopping of rsyslogd in an upstart system removes the /dev/log socket, which causes the swift service's attempt to log a message to fail - which it then proceeds to attempt to log, in the syslog. This spirals out and causes the program to crash due to the infinite recursion overflowing the call stack.

[0] https://github.com/openstack/swift/commit/95efd3f9035ec4141e1b182516f040a59a3e5aa6

This patch is for the xenial version of swift.

This patch is for the trusty-mitaka version of swift for the trusty-mitaka Ubuntu Cloud Archive

This patch is for the kilo version of swift for the trusty-kilo Ubuntu Cloud Archive.

This patch is for the icehouse version of swift in Trusty.

Confirmed this is already fixed in yakkety and later. Reviewed and sponsored for xenial and trusty.

The patches look superfluously different between Trusty and Xenial, and there are no dep3 headers in the Trusty patch. Was this backported, and if so by whom? Are they both separate cherry-picks from upstream, or otherwise what has been done to make the patch for Trusty work?

@Robie - sorry the trusty patch was indeed messed up. I've rebuilt and retested the patch with appropriate dep3 headers and such and you'll find it more inline with the other patch versions. The trusty one varies slightly due to the unit test import differences (I tried to keep the same style) and the method signature for logging in the trusty version.

15:07 <rbasak> wolsen: around? You need a sponsor for bug 1683076 I think?

15:08 <rbasak> wolsen: I suggest three minor changes to your trusty debdiff: https://git.launchpad.net/~racb/ubuntu/+source/swift/log/?h=lp1683076-trusty

15:09 <rbasak> The dep3 changes apply to your xenial debdiff too I think. But I'll accept without if you wish.

Can you find a sponsor for the Trusty debdiff please, and then I can accept the SRU?

Hey Robie, here's an updated patch for Xenial. I'll update the patch for Trusty next and find an additional sponsor to review it.

Updated Origin field to in patch to indicate the patch for xenial is a clean cherry-pick (e.g. upstream instead of backport per DEP-3 guidelines).

Here's a patch for trusty with updated DEP-3 headers and white space change log removal. I'll seek an additional sponsor for trusty per:

 <rbasak> wolsen: since you've rewritten the Trusty patch I don't feel that I can accept it myself as the SRU reviewer - it needs another sponsor +1.

Updated trusty-kilo patch with corrected DEP-3 headers

Updated trusty-mitaka patch with corrected DEP-3 headers

Thanks Billy. I've uploaded to trusty (unapproved queue), kilo-staging, and xenial (unapproved queue).

Hello Jill, or anyone else affected,

Accepted swift into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/swift/1.13.1-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Jill, or anyone else affected,

Accepted swift into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/swift/2.7.1-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Jill, or anyone else affected,

Accepted swift into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I have completed the verification of this bug fix for trusty-proposed and xenial-proposed packages in the Ubuntu project. Tests involve stopping the logging services (e.g. rsyslog) and ensuring that the services continue to run.

I have also completed the verification of this bug fix for the trusty-mitaka cloud archive proposed pocket. I have not seen the update hit the trusty-kilo cloud archive proposed pocket so will follow up with the Ubuntu OpenStack team on that.

Hello Jill, or anyone else affected,

Accepted swift into kilo-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:kilo-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-kilo-needed to verification-kilo-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-kilo-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The verification of the Stable Release Update for swift has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package swift - 2.7.1-0ubuntu2

---------------
swift (2.7.1-0ubuntu2) xenial; urgency=medium

  * Fix issue where swift daemons crash while writing logs to a stopped
    rsyslogd /dev/log socket. (LP: #1683076)
    - d/patches/fix-infinite-recursion-logging.patch: Cherry-picked from
      upstream stable/newton branch to avoid infinite loops when logging
      while rsyslogd is stopped.

 -- Billy Olsen <email address hidden> Mon, 22 May 2017 12:58:01 -0700

This bug was fixed in the package swift - 1.13.1-0ubuntu1.3

---------------
swift (1.13.1-0ubuntu1.3) trusty; urgency=medium

  * Fix issue where swift daemons crash while writing logs to a stopped
    rsyslogd /dev/log socket. (LP: #1683076)
    - d/patches/fix-infinite-recursion-logging.patch: Cherry-picked from
      upstream stable/newton branch to avoid infinite loops when logging
      while rsyslogd is stopped.

 -- Billy Olsen <email address hidden> Mon, 03 Jul 2017 22:22:58 -0700

I was able to finish verification for trusty-proposed/kilo cloud archive pocket this evening. Stop rsyslogd while swift services are running and verify the {object,account,container}-replicator services do not die due to the inability to write to the logs.

The verification of the Stable Release Update for swift has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package swift - 2.2.2-0ubuntu1.3~cloud1
---------------

 swift (2.2.2-0ubuntu1.3~cloud1) trusty-kilo; urgency=medium
 .
   * Fix issue where swift daemons crash while writing logs to a stopped
     rsyslogd /dev/log socket. (LP: #1683076)
     - d/patches/fix-infinite-recursion-logging.patch: Cherry-picked from
       upstream stable/newton branch to avoid infinite loops when logging
       while rsyslogd is stopped.

The verification of the Stable Release Update for swift has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package swift - 2.7.1-0ubuntu2~cloud0
---------------

 swift (2.7.1-0ubuntu2~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 swift (2.7.1-0ubuntu2) xenial; urgency=medium
 .
   * Fix issue where swift daemons crash while writing logs to a stopped
     rsyslogd /dev/log socket. (LP: #1683076)
     - d/patches/fix-infinite-recursion-logging.patch: Cherry-picked from
       upstream stable/newton branch to avoid infinite loops when logging
       while rsyslogd is stopped.

Was the behavior of rsyslog prior to this fix as described in 1683075,
by any chance? Did the resolution wrt the storage issue affect the rsyslog
frequent stoppage? I'm trying to determine if 1683075 is still an issue..