Comment 14 for bug 1382556

Revision history for this message
Dave Cheney (dave-cheney) wrote : Re: [Bug 1382556] Re: "cannot allocate memory" when running "juju run"

> 3. forks should be using COW, but there may be Go runtime (or Juju) impact on the realized benefit

This is correct, but from the kernel's point of view the child process
_may_ touch every page it interpreted so the kernel accounts for the
child as 100% of the cost of the parent. The accounting is affected by
the amount of swap available, and the kernel's overcommit setting.

On Tue, Nov 10, 2015 at 11:56 AM, Eric Snow <email address hidden> wrote:
> >From what I can tell there are 2 different issues here:
>
> 1. memory/goroutine leaks (e.g. see https://www.vividcortex.com/blog/2014/01/15/two-go-memory-leaks/)
> 2. external commands are run via a fork
>
> For the leaks, we should make the effort to track down all leaks and fix
> them, regardless of how they manifest.
>
> For the external commands, there are several considerations:
>
> 1. exec.Cmd.Start() (and Run) is a light wrapper around os.StartProcess()
> 2. os.StartProcess() is a wrapper around syscall.ForkExec()
> 3. forks should be using COW, but there may be Go runtime (or Juju) impact on the realized benefit
> 4. in core exec.Cmd.Start() is called (at least indirectly) through juju/utils/exec.go, juju/utils/ssh, and utils/exec
> 5. the "forker" stategy John Meinel proposed on the list should help if done early in jujud's main
> 6. updating juju/utils/ssh to use the forker daemon would have the biggest immediate impact
>
> I spent some time today looking into a reasonable approach to the forker
> strategy, with special consideration for the juju/utils/ssh package.
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> Matching subscriptions: MOAR JUJU SPAM!
> https://bugs.launchpad.net/bugs/1382556
>
> Title:
> "cannot allocate memory" when running "juju run"
>
> Status in juju-core:
> In Progress
> Status in juju-core 1.25 series:
> In Progress
>
> Bug description:
> I executed:
> juju run "uptime" --all
>
> and is returned messages like this for every machine:
> - Error: 'fork/exec /usr/bin/ssh: cannot allocate memory'
> MachineId: "9"
> Stdout: ""
>
> I can "juju ssh" to my instances fine and they all have memory
> available.
>
> According to the documentation above command should be fine:
> https://github.com/juju/cheatsheet
> "juju run "uname -a" --all - Run a command (uname) on every instance. (1.17.2 and newer)"
>
> I am running juju 1.18.4
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1382556/+subscriptions