[SRU] nova-* services do not start if rsyslog is not yet started

The attachment "nova-2014.1.4-0ubuntu3-lp1459046.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Don't each of these start lines need a different WAITER value?

Michael,

Yeah, they do. Thanks a lot for pointing it out! I am going to have a thorough testing, and will update the patch soon.

Thanks,
Liang

This is apparently an SRU fix for trusty. Does that also affect later releases? In particular, does it affect vivid/wily, as we don't use upstart there but systemd? In particular, these would need an After=syslog.service, is that the case? If not, this needs to be fixed as well, and as per the SRU rules a bug has to be fixed in the development series first. Thanks!

Hi Martin,

Yeah. This an SRU fix for trusty. It doesn't affect later releases, as newer releases use openstack juno/kilo which doesn't have this problem.
That reminded me to try a backport instead of working on a upstart job patch. And it's indeed simpler and smaller. Thanks!

Thanks,
Liang

Hi Liang

Two comment; firstly your cherry pick is not quite clean - it retains the existing syslog config code after the try/except wrapper introduced upstream.

Secondly I'm not sure that this really resolves the issue - if syslog is not started prior to a nova service, nova will just not use syslog for logging - which feels wrong to me.

I think a better fix might be to retry configuring syslog a few times - that way we don't have to update all upstart configurations, as the common code in the nova codebase will deal with the race.

Removing sponsors for now - lets get the fix right and then go for sponsorship.

Hi James,

Sorry, I got messed up with quilt. I have resubmit the SRU at https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1348244, as it has the major fix that I backported in this SRU. And along with the backport, I also implemented the retry logic as you suggested. Thank you. I will mark this bug as invalid and continue the SRU in the original bug.

Thanks,
Liang

The patch you have provided here for SRU (that is also now part of the SRU submission in bug 1348244) does not appear to have been submitted upstream. So, does this affect openstack versions greater than Icehouse? It looks like Nova logging was switched to use oslo.log as of Kilo so presumably Juno is also affected by this?

Assuming that Kilo+ is not affected and since this code has been moved out of tree anyway I think we should apply this to both I and J and make sure that patch commit message references this bug so that we have a context reference.

stable/icehouse patch uploaded to https://review.openstack.org/#/c/193105/

stable/juno patch uploaded to https://review.openstack.org/#/c/193110/

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/193214

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/193215

Kilo & master status:

Master works without any problem. oslo.log 1.2.0 will start writing to syslog as soon as rsyslogd is brought up.
Kilo uses oslo.log 1.1.0 which just ignores use_syslog setting when rsyslogd is not running or not started before nova-* services.

I have proposed the backport (https://review.openstack.org/#/c/193633/) to oslo.log stable/kilo from oslo.log master, and will get to the maintainer do a 1.0.1 release once merged.

Reviewed: https://review.openstack.org/193633
Committed: https://git.openstack.org/cgit/openstack/oslo.log/commit/?id=9a6c22663b57b7f2d9071767dadf28e59075a75e
Submitter: Jenkins
Branch: stable/kilo

commit 9a6c22663b57b7f2d9071767dadf28e59075a75e
Author: Julien Danjou <email address hidden>
Date: Mon Nov 24 18:35:36 2014 +0100

    Replace RFCSysLogHandler by a syslog() based one

    Rather than using the Python provided logging module that reimplents the
    whole syslog protocol and pointing it to /dev/log (which is not
    portable), we use the system calls from the syslog module, which are
    portable and default to the behaviour we actually want.

    Closes-Bug: 1385295
    Closes-Bug: 1391428

    Partial-Bug: 1459046

    (cherry picked from commit 33f5c6f94d94d5464fea38229d880f177b05aff7)

    Conflicts:
     oslo_log/log.py
     requirements.txt

    Change-Id: I2e47c896841cbd43d1f5504f2b8b897fc432cd0b

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193105
Reason: This branch has reached end of life and is being deleted.

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193209
Reason: This branch has reached end of life and is being deleted.

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193214
Reason: This branch has reached end of life and is being deleted.

Change abandoned by Jeremy Stanley (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/193215
Reason: This branch has reached end of life and is being deleted.

Change abandoned by Edward Hope-Morley (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/193110

Confirmed as fixed in Wily (oslo.log 1.2.0)

As Utopic goes EOL this month, marking Utopic task as 'Won't Fix' - we will target this directly to the Cloud Archive for Juno

Raised bug tasks for python-oslo.log, as we have fix to land for vivid/kilo

Hello Liang, or anyone else affected,

Accepted nova into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/1:2014.1.5-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The package uploaded to proposed has failed to build and I see a lot of these messages in the buildlog:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/sphinx/ext/autodoc.py", line 335, in import_object
    __import__(self.modname)
  File "/«PKGBUILDDIR»/nova/tests/__init__.py", line 37, in <module>
    % os.environ.get('EVENTLET_NO_GREENDNS'))
ImportError: eventlet imported before nova/cmd/__init__ (env var set to None)

fwiw i just tried building this myself and it succeeded so not sure why the proposed pocket fails:

https://launchpad.net/~hopem/+archive/ubuntu/trusty-sru-testing-lp1459046/+build/7667621

Turns out the problem here a dodgy python-oslo.messaging in proposed from bug 1362863. This has now been removed so hopefully a re-spin will succeed.

New proposed build was successful. I have now deployed and tested that this work as expected.

This bug was fixed in the package nova - 1:2014.1.5-0ubuntu1.2

---------------
nova (1:2014.1.5-0ubuntu1.2) trusty; urgency=medium

  * Add rsyslog retry support (LP: #1459046)
    - d/p/add-support-for-syslog-connect-retries.patch
  * Add vm clean shutdown support (LP: #1196924)
    - d/p/clean-shutdown.patch

 -- Edward Hope-Morley <email address hidden> Thu, 16 Jul 2015 11:55:57 +0100

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Fixed in oslo.log (which Nova uses)