juju destroy-environment -y local --force does not stop jujud/mongod processes on broken bootstrap

top - 00:06:40 up 2 days, 23:09, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 82 total, 1 running, 76 sleeping, 5 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si,
 0.0 st
KiB Mem: 8255588 total, 1005912 used, 7249676 free, 88848 buffers
KiB Swap: 0 total, 0 used, 0 free. 685768 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

15310 ubuntu 20 0 370348 78212 12620 T 0.0 0.9 0:10.98 juju

15368 root 20 0 403920 34504 11592 T 0.0 0.4 0:00.16 jujud

^ T state, ooh err.

T state means someone sent STOP to the process, or it's being traced (unlikely)

Possibly the shutdown code is sending -SIGSTOP not SIGTERM or SIGKILL. Possibly this is coming from upstart rather than the local provider.

I think, actually, that Upstart uses ptrace for processes it spawns so that it can see when that process spawns children:
https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-upstart-overcome-ptrace-limitations

So probably the T is because it actually *is* being traced.

oh. my. god. That is going to end terribly. Is there is a switch to disable
that ?

On Mon, Mar 17, 2014 at 3:25 PM, John A Meinel <email address hidden>wrote:

> I think, actually, that Upstart uses ptrace for processes it spawns so
> that it can see when that process spawns children:
>
> https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-upstart-overcome-ptrace-limitations
>
> So probably the T is because it actually *is* being traced.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1292275
>
> Title:
> juju destroy-environment -y local --force does not stop jujud
> processes
>
> Status in juju-core:
> Triaged
>
> Bug description:
> Situation.
>
> $ juju bootstrap -e local --upload-tools
>
> failed because /usr/bin/mongod is not installed, see LP #1271937
>
> $ juju destroy-environment -y local
>
> fails because the environment is not bootstrapped
>
> ERROR state/api: websocket.Dial wss://10.0.3.1:17070/: dial tcp
> 10.0.3.1:17070: connection refused
> ERROR state/api: websocket.Dial wss://10.0.3.1:17070/: dial tcp
> 10.0.3.1:17070: connection refused
> ERROR state/api: websocket.Dial wss://10.0.3.1:17070/: dial tcp
> 10.0.3.1:17070: connection refused
> ERROR state/api: websocket.Dial wss://10.0.3.1:17070/: dial tcp
> 10.0.3.1:17070: connection refused
> ERROR state/api: websocket.Dial wss://10.0.3.1:17070/: dial tcp
> 10.0.3.1:17070: connection refused
> ERROR state/api: websocket.Dial wss://10.0.3.1:17070/: dial tcp
> 10.0.3.1:17070: connection refused
> ERROR state/api: websocket.Dial wss://10.0.3.1:17070/: dial tcp
> 10.0.3.1:17070: connection refused
>
> $ juju destroy-environment -y local --force
>
> works, but does not stop the jujud processes
>
> ubuntu@winton-02:~/src/launchpad.net/juju-core$ pstree
> init─┬─acpid
> ├─atd
> ├─cron
> ├─dbus-daemon
> ├─dnsmasq
> ├─7*[getty]
> ├─irqbalance
> ├─rsyslogd───3*[{rsyslogd}]
> ├─sshd───sshd───sshd───bash───tmux
> ├─systemd-logind
> ├─systemd-udevd
> ├─tmux─┬─bash───vim───{vim}
> │ └─bash─┬─juju─┬─sudo───bash───bash───jujud───6*[{jujud}] //
> oops
> │ │ └─6*[{juju}]
> │ └─pstree
> ├─upstart-file-br
> ├─upstart-socket-
> └─upstart-udev-br
>
> ubuntu@winton-02:~/src/launchpad.net/juju-core$ pgrep -lf jujud
> 15368 jujud
> ubuntu@winton-02:~/src/launchpad.net/juju-core$ sudo !!
> sudo pgrep -lf jujud
> 15368 jujud
> 15516 sudo
> ubuntu@winton-02:~/src/launchpad.net/juju-core$ cat /proc/15368/cmdline
>
> /home/ubuntu/.juju/local/tools/1.17.5.1-trusty-ppc64/jujudbootstrap-state--data-dir/home/ubuntu/.juju/local--env-configYWRtaW4tc2VjcmV0OiAiIgphZ2VudC12ZXJzaW9uOiAxLjE3LjUuMQphcGktcG9ydDogMTcwNzAKYXB0LWZ0cC1wcm94eTogIiIKYXB0LWh0dHAtcHJveHk6ICIiCmFwdC1odHRwcy1wcm94eTogIiIKYXV0aG9yaXplZC1rZXlzOiAnc3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFEQ2EzL2lDRHBwQkZQemlZbzdqblJDdFRwRkdLdnppQm1FKzloaDFiUnd3eXNjanczTkR5UmxWY2NJTFB1ZVczS3BicG5ZOEx3d3BRN3FhVVNscVFGcTkyTnRGUEdhYTJlMndGSVFIYnlxc1VQQ1B4dEY1d1lxUVhYY0...

The termination code sends SIGABRT (slightly abusive, but SIGTERM is used by upstart).

The problem here is that since mongod doesn't exist, the bootstrap agent never finishes. The agent never gets as far as starting the signal-handler worker, hence why termination/uninstall doesn't work.

Since this only happened on a momentarily broken trunk (trunk is fixed now, you can't bootstrap without mongod anymore), I'm not convinced there's anything to do.

BTW the bootstrap agent is run directly, not through upstart. Seems likely to me that it just got ctrl-z'd.

John, I don't think there's anything to do here for now. If the bootstrap agent hangs then something is fundamentally broken, and requires user intervention.

There's argument to be made for having a more intelligent destroy-env to handle this and similarly broken cases, but I think this should be marked Low and have its milestone removed. I'll wait for you before doing that.

I've just confirmed that destroy-environment --force *does* kill the bootstrap agent, but obviously not if the process is stopped (where the signal won't be acted upon until the process wakes up). I'm closing this, if someone disagrees then they can reopen.

I've just seen that it leaves mongodb running, so I'm going to reopen this.
I'm not sure how I got into this situation, but I bootstrapped the local provider, upgraded it, killed machine-0 agent a few times, and in the end, I have a mongod running that I had to kill manually.

I don't think this has to block 1.17.6, but it does still seem to be a bug.

Note that I do still have:
$ ls /etc/init/juju*
/etc/init/juju-agent-jameinel-local.conf /etc/init/juju-db-jameinel-local.conf

However, those are now listed as "stop/waiting", though that was after I manually killed mongod (upstart didn't seem to want to auto-restart it, which surprises me a bit)