Trying to bootstrap a juju 2.0.0 controller on Xenial LXD host. Command and error message:
$ juju bootstrap --config juju2-bootstrap.yaml lxd lxd-xenial --debug
[...]
2016-10-15 23:02:54 INFO juju.cmd supercommand.go:63 running jujud [2.0.0 gc go1.6.2]
2016-10-15 23:02:54 DEBUG juju.cmd supercommand.go:64 args: []string{"/var/lib/juju/tools/2.0.0-xenial-amd64/jujud", "bootstrap-state", "--timeout", "20m0s", "--data-dir", "/var/lib/juju", "--debug", "/var/lib/juju/bootstrap-params"}
2016-10-15 23:02:54 DEBUG juju.agent agent.go:509 read agent config, format "2.0"
2016-10-15 23:02:54 DEBUG juju.tools.lxdclient client.go:199 connecting to LXD remote "remote": "10.10.1.254:8443"
2016-10-15 23:02:54 ERROR cmd supercommand.go:458 new environ: creating LXD client: Get https://10.10.1.254:8443/1.0: Unable to connect to: 10.10.1.254:8443
2016-10-15 23:02:54 DEBUG cmd supercommand.go:459 (error details: [{github.com/juju/juju/cmd/jujud/bootstrap.go:144: new environ} {github.com/juju/juju/provider/lxd/provider.go:32: } {github.com/juju/juju/provider/lxd/environ.go:59: } {github.com/juju/juju/provider/lxd/environ_raw.go:71: creating LXD client} {github.com/juju/juju/provider/lxd/environ_raw.go:107: } {github.com/juju/juju/tools/lxdclient/client.go:124: } {github.com/juju/juju/tools/lxdclient/client.go:241: } {Get https://10.10.1.254:8443/1.0: Unable to connect to: 10.10.1.254:8443}])
[...]
The issue seems to be that the agent running in the LXD container being created uses the gateway's IP (10.10.1.254) instead of the LXD host's (10.10.1.80). This is a clean juju-2 installation.
Connections to outside internet work fine (through the gateway and http/apt proxy). I can isolate the issue to juju with the following little change (at the gateway machine): redirecting the port 8443 back to the LXD host from the gateway (masq/DNAT) allows the bootstrap to proceed and results in a working controller (LXD container) that can install charms. This way, traffic is bounced through the gateway which obviously is wrong. Removing the DNAT rules stops the controller from working after bootstrap.
There seems to be no way to specify the LXD host's address by hand. Referring to this pull: https://github.com/juju/juju/pull/6078 (info from bug https://bugs.launchpad.net/juju/+bug/1618636 ). However, the suggested workaround there is wrong/incomplete as I already have https_address set in LXD and the whole shebang works if I redirect port 8443 back from gateway.
---
Configuration (on LXD host machine):
$ ip -4 address show up
6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65500 qdisc pfifo_fast state UP
inet 10.10.10.80/24 brd 10.10.10.255 scope global ib0
10: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
inet 10.10.1.80/24 brd 10.10.1.255 scope global br-int
(br-int is an openvswitch bridge)
$ ip r
default via 10.10.1.254 dev br-int onlink
10.10.1.0/24 dev br-int proto kernel scope link src 10.10.1.80
10.10.3.2 via 10.10.1.254 dev br-int proto zebra metric 20
10.10.9.0/24 via 10.10.1.254 dev br-int proto zebra metric 20
10.10.10.0/24 dev ib0 proto kernel scope link src 10.10.10.80
$ cat juju2-bootstrap.yaml
http-proxy: http://10.10.1.254:8080
https-proxy: http://10.10.1.254:8080
apt-http-proxy: http://10.10.1.254:3142
apt-https-proxy: http://10.10.1.254:3142
no-proxy: localhost,10.10.1.254,10.10.1.80
$ groups
USERNAME adm disk cdrom sudo dip plugdev libvirtd lpadmin sambashare lxd
$ lxc info
apiextensions: []
apistatus: stable
apiversion: "1.0"
auth: trusted
environment:
addresses:
- 10.10.10.80:8443
- 10.10.1.80:8443
architectures:
- x86_64
- i686
certificate: [...]
driver: lxc
driverversion: 2.0.4
kernel: Linux
kernelarchitecture: x86_64
kernelversion: 4.4.0-38-generic
server: lxd
serverpid: 17369
serverversion: 2.0.4
storage: zfs
storageversion: "5"
config:
core.https_address: '[::]'
core.proxy_http: http://10.10.1.254:8080
core.proxy_https: http://10.10.1.254:8080
core.proxy_ignore_hosts: localhost,10.10.1.254,10.10.1.80
core.trust_password: true
images.auto_update_interval: "24"
images.remote_cache_expiry: "14"
storage.zfs_pool_name: data/lxd
public: false
$ apt-cache policy juju
juju:
Installed: 1:2.0.0-0ubuntu1~16.04.2~juju1
Candidate: 1:2.0.0-0ubuntu1~16.04.2~juju1
Version table:
*** 1:2.0.0-0ubuntu1~16.04.2~juju1 100
100 /var/lib/dpkg/status
2.0~beta15-0ubuntu2.16.04.1 500
500 http://XX.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
500 http://XX.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
2.0~beta4-0ubuntu2 500
500 http://XX.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
500 http://XX.archive.ubuntu.com/ubuntu xenial/main i386 Packages
This issue completely prevents me from setting up a juju-2.0.0 controller on LXD. Looking at the code (now that I had some time) reveals clearly that the new (fallback?) mechanism is the reason for the wrong address being used:
https:/
Are bridges, where the LXD host itself is handling the subnetting (and is the default gateway) the only ones supported now? If not, there needs to be a way to define the host address manually, or the LXD address resolver needs to be fixed.