Ubuntu
upstart package

100% CPU usage in init if /dev/console is not available

Bug #880049 reported by Cedric Schieli on 2011-10-22

This bug affects 16 people

Affects		Status	Importance	Assigned to	Milestone
	upstart (Ubuntu)	Confirmed	Undecided	Unassigned

Bug Description

When /dev/console is not available (for example when running as an OpenVZ guest), init consumes 100% time of one CPU core.
Here is a strace excerpt captured on the OpenVZ host :

...
waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {3903451, 826182267}) = 0
select(5, [0 3 4], [], [4], NULL) = 1 (in [0])
read(0, "", 1) = 0
waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {3903451, 826467596}) = 0
select(5, [0 3 4], [], [4], NULL) = 1 (in [0])
read(0, "", 1) = 0
waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {3903451, 827157279}) = 0
select(5, [0 3 4], [], [4], NULL) = 1 (in [0])
read(0, "", 1) = 0
...

Here is what happens :

- in init's main function, system_setup_console(CONSOLE_OUTPUT) fails, leaving file descriptors 0 and 1 uninitialized
- nih_main_loop_init eventually get called, initializing its interrupt_pipe on file descriptors 0 and 1
- we end up calling system_setup_console(CONSOLE_NONE), resetting file descriptors 0 and 1 to /dev/null
- in nih_main_loop, select is called on 0 (/dev/null instead of the pipe) which repeatedly appears to have changed

I'm attaching a small patch which fixes this.

Related branches

lp://staging/~jammy/ubuntu/oneiric/upstart/upstart.fix-880049

Revision history for this message

Cedric Schieli (cschieli) wrote on 2011-10-22:

Reopen /dev/null if opening /dev/console failed Edit (649 bytes, text/x-diff)

Revision history for this message

Launchpad Janitor (janitor) wrote on 2011-10-24:

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in upstart (Ubuntu):
status:	New → Confirmed

Revision history for this message

davidlyness (davidlyness) wrote on 2011-10-24:

I am affected by this and do not have the technical knowledge to modify and rebuild the upstart source code. Is there a fix that does not require altering the source, or is this likely to make it into an update some time soon?

Revision history for this message

NoahY (noahy) wrote on 2011-10-25:

I have the same basic problem, but the patch given doesn't do anything. Trying installing upstart 1.6.7, no better. init is always at 100% cpu.

Revision history for this message

James Hunt (jamesodhunt) wrote on 2011-10-25:

Thanks for reporting this issue. However, the console device is a device expected to exist on all Linux systems. The problem here is that Upstart *needs* access to the console: if it has no console access, jobs would behave differently in an OpenVZ container to running outside a container (bad). The patch above will not help jobs that have specified they want console access or ownership ("console output" or "console owner").

I am not an OpenVZ expert, but I do feel that the "fix" for this issue is for the OpenVZ environment to provide a /dev/console that Upstart is able to use such as would be provided by other virtualization technologies (LXC, qemu/kvm, Xen, VirtualBox) otherwise Upstart cannot guarantee job behaviour.

Revision history for this message

Cedric Schieli (cschieli) wrote on 2011-10-25:

I totally agree the proper fix would be that OpenVZ provides a /dev/console. My patch is rather a workaround.
Concerning jobs specifying console access, they already fallback to CONSOLE_NONE in job_process_spawn so they are working (maybe not exactly with the same behaviour though).
Concerning jobs specifying console ownership, they will fail indeed, but those jobs (rcS, mountall-shell, ...) really can't be used in a OpenVZ environment anyway.

Revision history for this message

NoahY (noahy) wrote on 2011-10-25:

Oh, I'm sorry for my stupidity... can confirm that the patch above works here - no more 100% cpu usage after rebooting.

Revision history for this message

software-schlosser (software-schlosser) wrote on 2011-10-25:

Just wanted to confirm the patch is working for me, many thanks! Hope to see it in upstream soon :)

Revision history for this message

Ajunne (ajunne) wrote on 2011-10-28:

I have the same issue here, physical server runs CentOS 6.0, VE runs Ubuntu.

You can easily grant access to the physical version of /dev/console with the following command on your physical server:

vzctl set 105 --devices c:5:1:rw --save

Replace 105 with the number of you VE.

After this, reboot and init will no longer be hogging 100% cpu.

Revision history for this message

Cedric Schieli (cschieli) wrote on 2011-10-28: Re: [Bug 880049] Re: 100% CPU usage in init if /dev/console is not available

#10

@Ajunne

Unfortunately that doesn't work for me.
After granting access to /dev/console to the VE (and rebooting it), I get
"No such device" instead of "Operation not permitted" when trying to access
the device and init still hogs the cpu.
Maybe it is a problem with the host (Debian 6.0,
linux-image-2.6.32-5-openvz-686 = 2.6.32-31) ? I'll investigate more deeply
in a few days.

Revision history for this message

James Hewitt (jammy) wrote on 2011-11-04:

#11

I would consider this a regression - it was working fine in Natty, so has been broken for oneiric.

I would like to see a default ubuntu instance work without changes in an openvz environment - it would help encourage the use of ubuntu for virtualized environments. For me, ubuntu wouldn't start at all until I had made some changes to the init scripts.

Revision history for this message

xtsbdu3reyrbrmroezob (xtsbdu3reyrbrmroezob) wrote on 2011-11-04:

#12

This is definitely still broken :(

Revision history for this message

xtsbdu3reyrbrmroezob (xtsbdu3reyrbrmroezob) wrote on 2011-11-04:

#13

The specified vzctl command does not fix anything...

Revision history for this message

ahmadshamli (ahmadshamli87) wrote on 2011-11-07:

#14

I'm using Parallel Virtuozzo. After upgrade from 11.04 to 11.10, init is using 100% of cpu all the time.

About the patch,how to use it.?

Revision history for this message

Cedric Schieli (cschieli) wrote on 2011-11-07:

#15

I found another workaround which do not involve patching : simply delete
/dev/console in the VE and replace it with a plain file.

Revision history for this message

xtsbdu3reyrbrmroezob (xtsbdu3reyrbrmroezob) wrote on 2011-11-07:

#16

On Mon, Nov 7, 2011 at 4:45 AM, Cedric Schieli <email address hidden> wrote:
> I found another workaround which do not involve patching : simply delete
> /dev/console in the VE and replace it with a plain file.

Doesn't work. VE will not boot if the file is a regular file...
--
Kristian Erik Hermansen
https://profiles.google.com/kristian.hermansen

Revision history for this message

2GooD (david+launchpad) wrote on 2011-11-12:

#17

It seems like there is a patch for this in the upstart trunk already?

http://bazaar.launchpad.net/~upstart-devel/upstart/trunk/revision/1326

Revision history for this message

Scott James Remnant (scott) wrote on 2011-11-12:

#18

Please don't subscribe people to bugs without their permission

Revision history for this message

2GooD (david+launchpad) wrote on 2011-11-13:

#19

I'm sorry Scott, I will not do that again. Thanks for your patience.

Revision history for this message

David Schoen (neerolyte) wrote on 2011-11-24:

#20

> On Mon, Nov 7, 2011 at 4:45 AM, Cedric Schieli <email address hidden> wrote:
> > I found another workaround which do not involve patching : simply delete
> > /dev/console in the VE and replace it with a plain file.
>
> Doesn't work. VE will not boot if the file is a regular file...

I was able to work around this by replacing /dev/console with a symlink to /dev/null and rebooting, i.e.:
# rm /dev/console
# cd /dev
# ln -s null console
# reboot

This was on something the vendor is calling a "Virtuozzo" which I think might just be OpenVZ under the hood.

Revision history for this message

James Hewitt (jammy) wrote on 2011-11-24:

#21

This has actually stopped happening on my server, after an upgrade by my hosting provider.

For those still affected by the issue, I have created a PPA that contains the latest oneiric package, with just this one upstream bugfix added:
https://code.launchpad.net/~jammy/+archive/upstart.fix-880049

Its waiting to build on launchpad now, packages should be available in a few hours.

Revision history for this message

Torsten Knodt (torsknod) wrote on 2011-11-26:

#22

I would like to add something to the opinion that openvz should fix this.
I agree that it should be fixed by openvz.
But many people are not in control of the openvz/ Virtuozzo installation, because they just have rent a virtual server. And as it worked before it should still work because else those affected people don't trust into the dependability of Ubuntu any more. And by the way, there are many fixes for bad hardware in the kernel, why shouldn't there be fixes in Ubuntu for bad virtual hardware.

Revision history for this message

xtsbdu3reyrbrmroezob (xtsbdu3reyrbrmroezob) wrote on 2011-12-02:

#23

On Wed, Nov 23, 2011 at 6:08 PM, David Schoen <email address hidden> wrote:
> I was able to work around this by replacing /dev/console with a symlink to /dev/null and rebooting, i.e.:
> # rm /dev/console
> # cd /dev
> # ln -s null console
> # reboot

Container will not boot (only mount) if the above is attempted. Not a
solution...
--
Kristian Erik Hermansen
https://profiles.google.com/kristian.hermansen