juju userdata should not restart networking

I think this is really a maas + lxc bug. I need to look into this further to understand where the fix really needs to be made.

Quoting Curtis Hovey (<email address hidden>):
> ** Summary changed:
>
> - Juju deploy of Charm in MAAS provider failing after successful bootstrap. Juju status stuck in “Pending” state
> + Juju deploy of Charm in MAAS fails because dbus fails
>

Based on what I see in the machine.log, it looks like dbus was not
installed. This is curious since AIUI juju uses the ubuntu-cloud
image. When I install a precise ubuntu-cloud container,
/var/run/dbus/system_bus_socket does exist. With ubuntu containers,
it does not until I 'apt-get install dbus.'

Could you please re-run this and, while the container is hung, log
in with 'sudo lxc-console -n <container-name>', and see whether the
dbus package is installed?

 status: incomplete

Sorry, I see now that you ssh in to start dbus, so obviously it's
installed.

 status: new

Status changed to 'Confirmed' because the bug affects multiple users.

Adding juju-core back because the bug still does affect functionality of juju-core and needs to be tracked in juju-core project, regardless of where/how the fix is applied.

I hit the same problem, and don't use LXC at all on my setup. I therefore will change this bug to point back to juju-core.

My MaaS lives in a KVM machine on my laptop.
It can PXE boot/start-stop other KVM machines living on the same NAT'ed network.
- "juju bootstrap --upload-tools" works (the node is spun up, and after the ubuntu install, "juju status" works).
- I can "juju deploy" any charm, however:
- The machines come up (the node boots, ubuntu gets installed), but "juju status" reports "pending" forever.

The machine that is brought up has the same symptoms as reported above:

The machine log in /var/log/juju show:
2013-11-05 18:39:02 ERROR juju runner.go:211 worker: exited "deployer": exec ["start" "--system" "jujud-unit-rabbitmq-server-0"]: exit status 1 (start: Unable to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory)

Running this command by hand works when run without "--system".

Once ran, /var/log/juju contains a unit log as:
http://pastebin.ubuntu.com/6371502/

The cloud-init log and cloud-init-output log are:
http://pastebin.ubuntu.com/6371297/
http://pastebin.ubuntu.com/6371310/

I further tried to fix the situation manually by symlinking "/var/lib/juju/tools/1.16.2.1-saucy-amd64/" to "/var/lib/juju/tools/unit-mysql-0", but to no avail.

Changing back to juju-core since this affects me, and I'm not using LXC containers at all.

I can confirm that I experience exactly the same issue using Juju with MAAS and nodes all under KVM.

What release series are you deploying to? As in, what gets installed on the MAAS nodes? Does deploying to a different series help at all?

Julian,

The series for all MAAS nodes being deployed is 13.10. I will check with an older release, but, there were also complaints about nodes being deployed with 13.04 also having this problem.

Julian, I've also be deploying 13.10. I can step back through the releases tomorrow when I'm at work if you'd like (I'm in Germany).

It would be very useful if you could deploy precise (12.04). It's what
we test with in the QA lab and there's no problems.

I suspect this is a server image issue, but let's confirm on that
release testing.

Thanks guys.

I do not hit this bug when using 12.04 nodes (even with a 13.10 MaaS server), so it seems likely to be a problem with the saucy server image indeed.
I'll do some more matrix testing with at least trusty and raring, to try to find where the regression was introduced.

What's the best place to assign this? It is likely that some users will run into this in the future, we should try to fix the saucy image.

<andreas> ah, there is another difference between my env and tribaal's
<andreas> I used the fast path installer, which basically downloads the cloud image and dumps it on the nodes
<andreas> instead of going through the debian installer
<andreas> so if using fast-path is the big difference, that would explain why it works in the cloud too, because it's the same image.

I can also confirm that using the fast path installer with 13.10 does not have any dbus failures associated with it. I can deploy a charm on my MAAS nodes (so far) without dbus crapping out on me.

MAAS on its own does not deploy a stopped dbus service in my testing; it only happens when juju is involved. I've isolated this to "service networking restart" done by juju using cloud-init userdata. I've updated the description. I think a fix for juju would be straightforward: don't do that; use more targeted "ifdown" and "ifup" as required instead. Note that the "ifdown" should be called before changing /etc/network/interfaces.

However, adding a dbus task, as it is arguable that dbus should not die in this case, and certainly not silently. A "Won't Fix" resolution for dbus dying might be acceptable, though, if it is not expected to survive all network interfaces being deconfigured. But a log message explaining would be nice.

Status changed to 'Confirmed' because the bug affects multiple users.

I'm hitting this problem inside Precise KVM nodes. The bottom of cloud-init-output.log has this:

+ start jujud-machine-3
jujud-machine-3 start/running, process 2244
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/handlers.py", line 807, in emit
    self._connect_unixsocket(self.address)
  File "/usr/lib/python2.7/logging/handlers.py", line 745, in _connect_unixsocket
    self.socket.connect(address)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
Logged from file __init__.py, line 116

The MAAS KVM node is on Saucy.

Setting dbus task to Low priority, for triage purposes. I think the primary fix that needs to happen here is in juju.

@racb

I don't see why this is anythign to do with juju.

ubuntu@cloud1:~$ sudo status dbus
dbus start/running, process 464
ubuntu@cloud1:~$ sudo restart networking
<4>init: Disconnected from system bus
<4>init: whoopsie main process ended, respawning
networking start/running
ubuntu@cloud1:~$ sudo status dbus
dbus stop/waiting

So quite simply,

1. dbus stops on "deconfiguring-networking"

2. restart networking issues deconfiguring-networking

3. dbus only starts on local-filesystems, so it never restarts.

Somehow dbus needs to be made to restart when networking restarts, or it simply should not shut down at deconfiguring-networking.

Ok, stgraber has set me straight - 'restart networking' is not supposed to be used. Use ifdown -a and ifup -a if needed.

For the record,

#ubuntu-devel: 02/11/14 23:10 <stgraber> hallyn: so tl;dr, they should use ifdown eth0 => 'cat > /etc/network/eth0.conf << EOF\niface br0 inet dhcp\n bridge_ports eth0\nEOF\n" => ifup br0, that'll DTRT

Please make the appropriate changes to juju. I am marking this bug invalid for dbus.

This bug was fixed in the package juju-core - 1.17.6-0ubuntu1

---------------
juju-core (1.17.6-0ubuntu1) trusty; urgency=medium

  * New upstream point release, including fixes for:
    - br0 not bought up by cloud-init with MAAS provider (LP: #1271144).
    - ppc64el enablement for juju/lxc (LP: #1273769).
    - juju userdata should not restart networking (LP: #1248283).
    - error detecting hardware characteristics (LP: #1276909).
    - juju instances not including the default security group (LP: #1129720).
    - juju bootstrap does not honor https_proxy (LP: #1240260).
  * d/control,rules: Drop BD on bash-completion, install bash-completion
    direct from upstream source code.
  * d/rules: Set HOME prior to generating man pages.
  * d/control: Drop alternative dependency on mongodb-server; juju now only
    works on trusty with juju-mongodb.
 -- James Page <email address hidden> Mon, 24 Mar 2014 16:05:44 +0000

saucy has seen the end of its life and is no longer receiving any updates. Marking the saucy task for this ticket as "Won't Fix".