juju agent using lxcbr0 address as apiaddress instead of juju-br0 breaks agents

I'm seeing this error on new deploys on juju 1.21.1-trusty-amd64 from the stable PPA. I've attached an extract of the machine log showing it changing the API addresses from the correct values to the wrong values.?field.comment=I'm seeing this error on new deploys on juju 1.21.1-trusty-amd64 from the stable PPA. I've attached an extract of the (redacted) machine log showing it changing the API addresses from the correct value (192.168.99.0) to the wrong value (10.0.3.1).

(Apologies for the repeated comment above - no idea what happened there.)

There are corresponding messages on machine 0, where it shows:

2015-02-03 01:17:36 INFO juju.worker.machiner machiner.go:92 setting addresses for machine-0 to ["local-machine:127.0.0.1" "local-cloud:192.168.99.0" "local-cloud:10.0.3.1" "local-machine:::1"]
...
2015-02-03 01:17:36 DEBUG juju.apiserver apiserver.go:156 <- [6D] machine-0 {"RequestId":21,"Type":"Machiner","Request":"SetMachineAddresses","Params":{"MachineAddresses":[{"Tag":"machine-0","Addresses":[{"Value":"127.0.0.1","Type":"ipv4","NetworkName":"","Scope":"local-machine"},{"Value":"192.168.99.0","Type":"ipv4","NetworkName":"","Scope":"local-cloud"},{"Value":"10.0.3.1","Type":"ipv4","NetworkName":"","Scope":"local-cloud"},{"Value":"::1","Type":"ipv6","NetworkName":"","Scope":"local-machine"}]}]}}

This happens only AFTER the bootstrap node is rebooted for the first time.

On a new env I was able to reboot all other nodes without a problem - apiaddress still points to the right IP of bootstrap node - provided the bootstrap node hadn't been rebooted after the first lxc was deployed.

By inspecting the agent.conf from LXC running on the bootstrap node I could see the apiaddress is pointing to the right ip BEFORE reboot.

AFTER the first reboot of bootstrap node, the apiaddress was changed to the lxcbr0 ip. Subsequent reboots of bootstrap node didn't change the behaviour.

Any subsequent reboot of other nodes (AFTER the bootstrap node was rebooted for the first time AFTER the first lxc was deployed) caused a change in the apiaddress to the lxcbr0 ip in all agents on the rebooted node. So seems to be something happening when the bootstrap node is rebooted for the first time.

This may be related to bug 1417308.

Per the duplicates:
manual provider does not work if lxc is already installed on the machine selected to be the state-server
Juju will fail to upgrade to 1.21.1 if a restart has happened.

We need some more information about how to reproduce this and I don't want it to block the 1.21.2 release until then.
I've synced up this with Alexis and Wes.

Per the duplicates:

A. start an instance in a cloud and install lxc on it. With the manual provider, attempt to bootstrap it.

B. The former juju-ci3 was bootstrapped with an older version of juju, the lxc was installed. the upgrade to 1.21.1 failed.

Thanks for the info!

I still think this shouldn't block 1.21.2, so I'm removing the milestone (because 1.21.3 is not yet available).

This job is bootstrapping a machine which has lxc installed. When it starts passing, we will now a fix works
    http://juju-ci.vapour.ws:8080/job/manual-deploy-trusty-ppc64/

We see evidence that the manual-provider test case was broken by a separate change. We can see 1.21 was passing the case until a recent backport of a feature from master. I will split the manual case from this bug.

The case of a working stack switching the apiaddresses to an lxcbr0 address can be seen with these steps with 1.21 on a real cloud.
1. juju bootstrap
2. deploy ubuntu
3. juju ssh 0 "sudo apt-get install -y lxc"
4. reboot -n
The agents will then fail

~$ sudo grep -A 1 apiaddresses /var/lib/juju/agents/machine-0/agent.conf
apiaddresses:
- 10.0.3.1:17070

$ sudo grep -A 1 apiaddresses /var/lib/juju/agents/machine-1/agent.conf
apiaddresses:
- 10.0.3.1:17070

This issue also happen when bootstrapping 1.20, installing lxc, reboot, then upgrade to 1.21. This is my command line for the fewest steps:
juju bootatrap
juju ssh 0 "sudo grep -A 1 apiaddresses /var/lib/juju/agents/machine-0/agent.conf"
  apiaddresses:
  - 10.185.145.253:17070
juju ssh 0 "sudo apt-get install -y lxc"
juju ssh 0 "sudo reboot -n"
juju ssh 0 "sudo grep -A 1 apiaddresses /var/lib/juju/agents/machine-0/agent.conf"
  apiaddresses:
  - 10.0.3.1:17070

I confirmed that manual-provider and 1.21 is indeed broken the same way. With lxc installed on the target state-server:
juju bootstrap
juju ssh 0 "sudo grep -A 1 apiaddresses /var/lib/juju/agents/machine-0/agent.conf"
  apiaddresses:
  - 10.0.3.1:17070

I've analyzed the problem and used the given steps to reproduce it.
The issue is due to the way we sort and pick cloud-local addresses for machines (incl. the api server). We should filter out any 10.0.x.y addresses we can see on the machine before considering them as usabled, taking into account the existence and contents of /etc/default/lxc-net (e.g. LXC_ADDR="10.0.3.1" and/or whatever address has the bridge specified in LXC_BRIDGE).

This should solve the problem and it could be easily ported to 1.20, 1.21, 1.22, and trunk. I'll start on the fix.

Fix for 1.21 proposed at https://github.com/juju/juju/pull/1616

Once approved, this fix will be forward ported to 1.22 and 1.23 (trunk).

Fix landed in 1.21, let's see how all the various CI jobs will react.

I'm assigning the forward porting of the same fix to 1.22 and trunk to James.
James, let's have a chat tomorrow about this.

The last failure was caused by a dirty machine. We found mongod from the failed runs running. After cleaning the machine, we can see fixed juju passed and it cleans up mongod.

I have the same behavior. After the reboot, it got the lxcbr0 IP address instead of the eth0 one.
I have Installed 1.20.14-0ubuntu1~14.04.1~juju1.
What if I manually change apiaddresses value in /var/lib/juju/agents/machine-0-lxc-2/agent.conf and reboot? That would solve the problem?
BTW, all services are running (sudo juju status | grep agent-state)

No Sacha, it won't help as they will get overwritten.

I'd suggest upgrading to 1.21.3 from the Juju stable releases PPA https://launchpad.net/~juju/+archive/ubuntu/stable

I updated to 1.21.3. That fixed the problem. :)

Another happy user :)