Juju 2.0.1 with LXD on localhost "provisioning error" "image not imported!"

Bug #1642385 reported by Halverneus
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Andrew Wilkins

Bug Description

Likely due to being behind a proxy. I am able to bootstrap successfully, however deploying any charm, from the store or locally, results in a provisioning error:

model:
  name: default
  controller: lxd
  cloud: localhost
  region: localhost
  version: 2.0.1
machines:
  "0":
    juju-status:
      current: down
      message: agent is not communicating with the server
      since: 16 Nov 2016 12:12:27-08:00
    instance-id: pending
    machine-status:
      current: provisioning error <----------
      message: image not imported! <----------
      since: 16 Nov 2016 12:12:27-08:00
    series: xenial
applications:
  mysql:
    charm: cs:~project-calico/xenial/mysql-0
    series: xenial
    os: ubuntu
    charm-origin: jujucharms
    charm-name: mysql
    charm-rev: 0
    exposed: false
    application-status:
      current: waiting
      message: waiting for machine
      since: 16 Nov 2016 12:11:12-08:00
    relations:
      cluster:
      - mysql
    units:
      mysql/0:
        workload-status:
          current: waiting
          message: waiting for machine
          since: 16 Nov 2016 12:11:12-08:00
        juju-status:
          current: allocating
          since: 16 Nov 2016 12:11:12-08:00
        machine: "0"

Proxy settings:
$ lxc config get core.proxy_http
http://10.100.0.240:8080
$ lxc config get core.proxy_https
http://10.100.0.240:8080
$ lxc config get core.proxy_ignore_hosts
127.0.0.1,10.0.5.0,10.0.5.1,10.0.5.2,10.0.5.3,...,10.0.5.252,10.0.5.253,10.0.5.254,10.0.5.255,10.0.0.0/8

$ lxc profile show default
name: default
config:
  environment.HTTP_PROXY: http://wall.ad.selinc.com:8080
  environment.HTTPS_PROXY: http://wall.ad.selinc.com:8080
  environment.NO_PROXY: 127.0.0.1,localhost,10.0.0.0/8,10.0.5.0,10.0.5.1,10.0.5.2,10.0.5.3,...,10.0.5.251,10.0.5.252,10.0.5.253,10.0.5.254,10.0.5.255
  environment.http_proxy: http://wall.ad.selinc.com:8080
  environment.https_proxy: http://wall.ad.selinc.com:8080
  environment.no_proxy: 127.0.0.1,localhost,10.0.0.0/8,10.0.5.0,10.0.5.1,10.0.5.2,10.0.5.3,...,10.0.5.251,10.0.5.252,10.0.5.253,10.0.5.254,10.0.5.255
description: ""
devices:
  eth0:
    mtu: "9000"
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic

Configuration used to bootstrap Juju:
$ cat config.yaml
default-series: xenial
apt-http-proxy: http://10.0.5.1:8000
apt-https-proxy: http://10.0.5.1:8000
http-proxy: http://10.100.0.240:8080
https-proxy: http://10.100.0.240:8080
no-proxy: 127.0.0.1,localhost,10.0.0.0/8,10.0.5.0,10.0.5.1,10.0.5.2,10.0.5.3,...,10.0.5.251,10.0.5.252,10.0.5.253,10.0.5.254,10.0.5.255

As mentioned, I'm not seeing any issue with bootstrapping. LXD network is lxdbr0 using the 10.0.5.0 network. On a similar note, any chance of supporting the 10.0.0.0/8 format (or something similar)?

Revision history for this message
Anastasia (anastasia-macmood) wrote :

We track Juju 2.x issues on "juju" launchpad project. Re-targeting...

no longer affects: juju-core
Changed in juju:
status: New → Triaged
importance: Undecided → High
milestone: none → 2.1.0
Revision history for this message
Anastasia (anastasia-macmood) wrote :

@Halverneus,

We have fixed couple of proxy related bugs since 2.0.1both in 2.0 series and 2.1. For example, the most recent one is bug # 1654591.

Could you please re-try with 2.1-rc2?

Changed in juju:
status: Triaged → Incomplete
importance: High → Undecided
milestone: 2.1-rc2 → none
Revision history for this message
Halverneus (halverneus) wrote :

I'll try it out in a few hours and let you know. Thanks!

Revision history for this message
Halverneus (halverneus) wrote :

Looks like I still have the same problem on 2.1-rc1-genericlinux-amd64. Juju GUI installs with no issue. Attempted to deploy from GUI and from CLI using 'default' and 'controller'.

My settings:
$ juju version
2.1-rc1-genericlinux-amd64

$ uname -a
Linux jerostreDT 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:55:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ lxc profile show default
config:
  environment.HTTP_PROXY: http://10.100.0.240:8080
  environment.HTTPS_PROXY: http://10.100.0.240:8080
  environment.NO_PROXY: 127.0.0.1,localhost,10.0.0.0/8
  environment.http_proxy: http://10.100.0.240:8080
  environment.https_proxy: http://10.100.0.240:8080
  environment.no_proxy: 127.0.0.1,localhost,10.0.0.0/8
description: ""
devices:
  eth0:
    mtu: "9000"
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  eth1:
    mtu: "9000"
    name: eth1
    nictype: macvlan
    parent: br0
    type: nic
name: default

$ cat config.yaml
apt-http-proxy: http://10.100.0.240:8080
apt-https-proxy: http://10.100.0.240:8080
http-proxy: http://10.100.0.240:8080
https-proxy: http://10.100.0.240:8080
no-proxy: 127.0.0.1,localhost,10.0.0.0/8,10.0.5.1,10.0.5.0

What I have tried with the most success:
$ juju bootstrap --config config.yaml localhost lxd
$ juju deploy cs:mysql
$ juju gui
ERROR Juju GUI is not available: GET https://10.0.5.221:17070/gui/daf5f4c6-aa7b-403f-82f2-b15e344e6d5b/: Get https://10.0.5.221:17070/gui/daf5f4c6-aa7b-403f-82f2-b15e344e6d5b/: Tunnel Connection Failed
$ juju destroy-controller --destroy-all-models lxd
$ juju bootstrap --config config.yaml localhost lxd
$ juju deploy -m controller cs:mysql
$ debug-log -m controller
machine-0: 06:44:35 DEBUG juju.worker.dependency "unit-agent-deployer" manifold worker started
machine-0: 06:44:35 DEBUG juju.worker.dependency "ssh-identity-writer" manifold worker started
machine-0: 06:44:35 DEBUG juju.worker.dependency "ssh-identity-writer" manifold worker stopped: <nil>
machine-0: 06:44:35 DEBUG juju.apiserver <- [8] machine-0 {"request-id":35,"type":"Resumer","version":2,"request":"ResumeTransactions","params":"'params redacted'"}
machine-0: 06:44:35 WARNING juju.cmd.jujud determining kvm support: INFO: /dev/kvm does not exist
HINT: sudo modprobe kvm_intel
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.8.0-34-generic/modules.dep.bin'
modprobe: FATAL: Module msr not found in directory /lib/modules/4.8.0-34-generic
: exit status 1
no kvm containers possible
machine-0: 06:44:40 ERROR juju.rpc error writing response: write tcp 127.0.0.1:17070->127.0.0.1:60410: write: connection reset by peer
machine-0: 06:50:30 WARNING juju.provisioner failed to start instance (image not imported!), retrying in 10s (3 more attempts)
machine-0: 06:51:00 WARNING juju.provisioner failed to start instance (image not imported!), retrying in 10s (2 more attempts)
machine-0: 06:51:31 WARNING juju.provisioner failed to start instance (image not imported!), retrying in 10s (1 more attempts)
machine-0: 06:51:42 ERROR juju.provisioner cannot start instance for machine "1": image not imported!

Revision history for this message
Halverneus (halverneus) wrote :

I also gave the following a try (inserting proxy into juju-default and juju-controller), but I didn't suspect it would work. I also included /etc/environment, where I set the proxy variables for my host. Any thoughts about anything I might have missed or should have done differently?

$ lxc profile show juju-default
config:
  boot.autostart: "true"
  environment.HTTP_PROXY: http://10.100.0.240:8080
  environment.HTTPS_PROXY: http://10.100.0.240:8080
  environment.NO_PROXY: 127.0.0.1,localhost,10.0.0.0/8
  environment.http_proxy: http://10.100.0.240:8080
  environment.https_proxy: http://10.100.0.240:8080
  environment.no_proxy: 127.0.0.1,localhost,10.0.0.0/8
  linux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tables
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  eth0:
    mtu: "9000"
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  eth1:
    mtu: "9000"
    name: eth1
    nictype: bridged
    parent: lxdbr0
    type: nic
  kvm:
    path: /dev/kvm
    type: unix-char
  mem:
    path: /dev/mem
    type: unix-char
  root:
    path: /
    type: disk
  tun:
    path: /dev/net/tun
    type: unix-char
name: juju-default

$ lxc profile show juju-controller
config:
  boot.autostart: "true"
  environment.HTTP_PROXY: http://10.100.0.240:8080
  environment.HTTPS_PROXY: http://10.100.0.240:8080
  environment.NO_PROXY: 127.0.0.1,localhost,10.0.0.0/8
  environment.http_proxy: http://10.100.0.240:8080
  environment.https_proxy: http://10.100.0.240:8080
  environment.no_proxy: 127.0.0.1,localhost,10.0.0.0/8
  security.nesting: "true"
description: ""
devices: {}
name: juju-controller

$ lxc config get core.proxy_http
http://10.100.0.240:8080

$ lxc config get core.proxy_https
http://10.100.0.240:8080

$ lxc config get core.proxy_ignore_hosts
127.0.0.1,localhost,10.0.0.0/8

$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/dart/bin:/usr/local/go/bin"
http_proxy=http://10.100.0.240:8080
https_proxy=http://10.100.0.240:8080
HTTP_PROXY=http://10.100.0.240:8080
HTTPS_PROXY=http://10.100.0.240:8080
no_proxy=127.0.0.1,localhost,10.0.0.0/8
NO_PROXY=127.0.0.1,localhost,10.0.0.0/8
EDITOR=/usr/bin/vim

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Halverneus, have you tried using Juju's proxy configuration?

```
p=http://10.100.0.240:8080
nop=127.0.0.1,localhost,10.0.0.0/8
juju bootstrap ... --config http-proxy=$p --config https-proxy=$p --config no-proxy=$nop
```

Revision history for this message
Halverneus (halverneus) wrote :

If it has the same results as...

$ cat config.yaml
apt-http-proxy: http://10.100.0.240:8080
apt-https-proxy: http://10.100.0.240:8080
http-proxy: http://10.100.0.240:8080
https-proxy: http://10.100.0.240:8080
no-proxy: 127.0.0.1,localhost,10.0.0.0/8,10.0.5.1,10.0.5.0
What I have tried with the most success:
$ juju bootstrap --config config.yaml localhost lxd
$ juju deploy cs:mysql

...from comment #4 (4&5 are from the same round of testing), then yes. I will give it a try with the command you suggested tomorrow, when I'm back at work, though. At least to see if there is any difference.

Revision history for this message
Halverneus (halverneus) wrote :

Updated this morning and it appears that I lost the ability to bootstrap at all. I have one other thought about what the new problem may be but I need to logout to determine if it is causing an issue. See my next comment for results.

$ juju --version
2.1-rc2-genericlinux-amd64

$ juju bootstrap --config no-proxy=$q --config http-proxy=$p --config https-proxy=$p --config apt-http-proxy=$p --config apt-https-proxy=$p localhost lxd
ERROR Get https://10.0.5.1:8443/1.0: Tunnel Connection Failed

$ juju bootstrap --config config.yaml localhost lxd
ERROR Get https://10.0.5.1:8443/1.0: Tunnel Connection Failed

$ unset http_proxy$ juju bootstrap localhost lxd
ERROR Get https://10.0.5.1:8443/1.0: Tunnel Connection Failed

$ unset https_proxy
$ unset no_proxy
$ unset HTTP_PROXY
$ unset HTTPS_PROXY
$ unset NO_PROXY

$ juju bootstrap localhost lxd
Creating Juju controller "lxd" on localhost/localhost
Looking for packaged Juju agent version 2.1-rc2 for amd64
No packaged binary found, preparing local Juju agent binary
To configure your system to better support LXD containers, please see: https://github.com/lxc/lxd/blob/master/doc/production-setup.md
Launching controller instance(s) on localhost/localhost...
ERROR failed to bootstrap model: cannot start bootstrap instance: can't get info for image 'juju/xenial/amd64': not found

$ q=localhost,127.0.0.1,10.0.0.0/8,10.0.5.0/24,10.0.5.1

$ juju bootstrap --config no-proxy=$q --config http-proxy=$p --config https-proxy=$p --config apt-http-proxy=$p --config apt-https-proxy=$p localhost lxd
Creating Juju controller "lxd" on localhost/localhost
Looking for packaged Juju agent version 2.1-rc2 for amd64
No packaged binary found, preparing local Juju agent binary
To configure your system to better support LXD containers, please see: https://github.com/lxc/lxd/blob/master/doc/production-setup.md
Launching controller instance(s) on localhost/localhost...
ERROR failed to bootstrap model: cannot start bootstrap instance: can't get info for image 'juju/xenial/amd64': not found

$ juju bootstrap --config config.yaml localhost lxd
Creating Juju controller "lxd" on localhost/localhost
Looking for packaged Juju agent version 2.1-rc2 for amd64
No packaged binary found, preparing local Juju agent binary
To configure your system to better support LXD containers, please see: https://github.com/lxc/lxd/blob/master/doc/production-setup.md
Launching controller instance(s) on localhost/localhost...
ERROR failed to bootstrap model: cannot start bootstrap instance: can't get info for image 'juju/xenial/amd64': not found

Revision history for this message
Halverneus (halverneus) wrote :

No luck. I'm running Wayland. I figured it might be something like how ssh-add doesn't work in Wayland, so I rebooted into X and got the exact same results.

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Re comment #7: sorry, I missed #4; they are indeed equivalent. We'll keep digging.

Changed in juju:
status: Incomplete → Triaged
status: Triaged → In Progress
importance: Undecided → High
assignee: nobody → Andrew Wilkins (axwalk)
milestone: none → 2.1.1
Revision history for this message
Halverneus (halverneus) wrote :

Thanks! If I can help in any way, let me know.

Revision history for this message
Andrew Wilkins (axwalk) wrote :

I can reproduce the issue with 2.1-rc2. Looking for a solution now.

Revision history for this message
Andrew Wilkins (axwalk) wrote :
Revision history for this message
John A Meinel (jameinel) wrote :

Also note, "no_proxy=...,10.0.0.0/8" won't do anything, as 'no_proxy' takes a list of domain suffixes (eg .google.com) or concrete IP addresses, but generally things don't support CIDR notation. I wish they did.

While *Juju* could support CIDR notation, that won't work when passing it to WGET on CURL or when configuring LXD with proxy information. So charms, etc still won't be using the right settings.

Andrew Wilkins (axwalk)
Changed in juju:
milestone: 2.1.1 → 2.1.0
status: In Progress → Fix Committed
Revision history for this message
Halverneus (halverneus) wrote :

Thanks! I'll test it out in a couple hours when I get into work. I've ran into the lack of CIDR notation support on some of my own Go projects. Being this close to release, it isn't a feature I would have time to implement or risk adding last minute, but for a future release can anyone think of any reason not to automatically exclude the configured network range, or make exclusion a configurable flag? At least within my company, all IP addresses to which we deploy are within our proxy. Otherwise, I do have a script that lists out every IP address in a network; though it is a cumbersome solution.

Revision history for this message
Halverneus (halverneus) wrote :

I managed to get it working after building a snap from changeset 1d44d9c. It did take some 'proxy magic', though. The config.yaml didn't require any changes at all. A note for anyone else that runs into the issue, it also requires that the terminal used to run the bootstrap/deploy command to have the proxy settings set and "export no_proxy=10.0.5.1". To access the Juju GUI, the IP address needs to be added after the machine comes up. If using $no_proxy AND $NO_PROXY, then both need set exactly the same before any of it works.

The issue I was experiencing has been fixed. Thanks!

Revision history for this message
Halverneus (halverneus) wrote :

Deploying single machines works fine, however, deploying bundles breaks on the hooks and 'juju ssh' doesn't work, no matter the SSH setting on the command line. Anyway to add no_proxy values to the configuration after bootstrapping or choosing IP addresses? I get the sense I might need to just configure with every IP in the network in the no-proxy configuration value. I'll try that when I get a chance.

Curtis Hovey (sinzui)
Changed in juju:
milestone: 2.1.0 → 2.1.1
Revision history for this message
Halverneus (halverneus) wrote :
Download full text (4.5 KiB)

Solution that seems to fix all of the issues (I'm not sure if there is a limit to the size of the no-proxy variable, but if I try to deploy OpenStack it will be a few times longer):

$ echo $http_proxy
http://10.100.0.240:8080

$ echo $https_proxy
http://10.100.0.240:8080

$ echo $no_proxy
10.0.5.1

$ cat config.yaml
apt-http-proxy: http://10.100.0.240:8080
apt-https-proxy: http://10.100.0.240:8080
http-proxy: http://10.100.0.240:8080
https-proxy: http://10.100.0.240:8080
no-proxy: 10.0.5.0,10.0.5.1,10.0.5.2,10.0.5.3,10.0.5.4,10.0.5.5,10.0.5.6,10.0.5.7,10.0.5.8,10.0.5.9,10.0.5.10,10.0.5.11,10.0.5.12,10.0.5.13,10.0.5.14,10.0.5.15,10.0.5.16,10.0.5.17,10.0.5.18,10.0.5.19,10.0.5.20,10.0.5.21,10.0.5.22,10.0.5.23,10.0.5.24,10.0.5.25,10.0.5.26,10.0.5.27,10.0.5.28,10.0.5.29,10.0.5.30,10.0.5.31,10.0.5.32,10.0.5.33,10.0.5.34,10.0.5.35,10.0.5.36,10.0.5.37,10.0.5.38,10.0.5.39,10.0.5.40,10.0.5.41,10.0.5.42,10.0.5.43,10.0.5.44,10.0.5.45,10.0.5.46,10.0.5.47,10.0.5.48,10.0.5.49,10.0.5.50,10.0.5.51,10.0.5.52,10.0.5.53,10.0.5.54,10.0.5.55,10.0.5.56,10.0.5.57,10.0.5.58,10.0.5.59,10.0.5.60,10.0.5.61,10.0.5.62,10.0.5.63,10.0.5.64,10.0.5.65,10.0.5.66,10.0.5.67,10.0.5.68,10.0.5.69,10.0.5.70,10.0.5.71,10.0.5.72,10.0.5.73,10.0.5.74,10.0.5.75,10.0.5.76,10.0.5.77,10.0.5.78,10.0.5.79,10.0.5.80,10.0.5.81,10.0.5.82,10.0.5.83,10.0.5.84,10.0.5.85,10.0.5.86,10.0.5.87,10.0.5.88,10.0.5.89,10.0.5.90,10.0.5.91,10.0.5.92,10.0.5.93,10.0.5.94,10.0.5.95,10.0.5.96,10.0.5.97,10.0.5.98,10.0.5.99,10.0.5.100,10.0.5.101,10.0.5.102,10.0.5.103,10.0.5.104,10.0.5.105,10.0.5.106,10.0.5.107,10.0.5.108,10.0.5.109,10.0.5.110,10.0.5.111,10.0.5.112,10.0.5.113,10.0.5.114,10.0.5.115,10.0.5.116,10.0.5.117,10.0.5.118,10.0.5.119,10.0.5.120,10.0.5.121,10.0.5.122,10.0.5.123,10.0.5.124,10.0.5.125,10.0.5.126,10.0.5.127,10.0.5.128,10.0.5.129,10.0.5.130,10.0.5.131,10.0.5.132,10.0.5.133,10.0.5.134,10.0.5.135,10.0.5.136,10.0.5.137,10.0.5.138,10.0.5.139,10.0.5.140,10.0.5.141,10.0.5.142,10.0.5.143,10.0.5.144,10.0.5.145,10.0.5.146,10.0.5.147,10.0.5.148,10.0.5.149,10.0.5.150,10.0.5.151,10.0.5.152,10.0.5.153,10.0.5.154,10.0.5.155,10.0.5.156,10.0.5.157,10.0.5.158,10.0.5.159,10.0.5.160,10.0.5.161,10.0.5.162,10.0.5.163,10.0.5.164,10.0.5.165,10.0.5.166,10.0.5.167,10.0.5.168,10.0.5.169,10.0.5.170,10.0.5.171,10.0.5.172,10.0.5.173,10.0.5.174,10.0.5.175,10.0.5.176,10.0.5.177,10.0.5.178,10.0.5.179,10.0.5.180,10.0.5.181,10.0.5.182,10.0.5.183,10.0.5.184,10.0.5.185,10.0.5.186,10.0.5.187,10.0.5.188,10.0.5.189,10.0.5.190,10.0.5.191,10.0.5.192,10.0.5.193,10.0.5.194,10.0.5.195,10.0.5.196,10.0.5.197,10.0.5.198,10.0.5.199,10.0.5.200,10.0.5.201,10.0.5.202,10.0.5.203,10.0.5.204,10.0.5.205,10.0.5.206,10.0.5.207,10.0.5.208,10.0.5.209,10.0.5.210,10.0.5.211,10.0.5.212,10.0.5.213,10.0.5.214,10.0.5.215,10.0.5.216,10.0.5.217,10.0.5.218,10.0.5.219,10.0.5.220,10.0.5.221,10.0.5.222,10.0.5.223,10.0.5.224,10.0.5.225,10.0.5.226,10.0.5.227,10.0.5.228,10.0.5.229,10.0.5.230,10.0.5.231,10.0.5.232,10.0.5.233,10.0.5.234,10.0.5.235,10.0.5.236,10.0.5.237,10.0.5.238,10.0.5.239,10.0.5.240,10.0.5.241,10.0.5.242,10.0.5.243,10.0.5.244,10.0.5.245,10.0.5.246,10.0.5.247,10.0.5.248,10.0.5.249,10.0.5.250,10.0.5.251,10.0.5.252,10.0.5.253,10.0.5.254,localhost,127.0.0....

Read more...

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Halverneus, thank you very much for confirming the fix.

As John mentions in #14, the standard $no_proxy doesn't support CIDRs; and we're intentionally just mirroring that. I'm a little reluctant to expand them just because users might then get confused when Juju supports one format and every other tool supports another. OTOH I can see the value, so we'll consider it.

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Just to set expectations: this didn't make it in time for 2.1.0. The fix will be released in 2.1.1, which will be available in a couple of weeks.

Revision history for this message
Halverneus (halverneus) wrote :

No worries. I really appreciate the your work on this! It gives me something to work with to start momentum at work. Besides, I have a working snap and I can always apply your changeset as a patch on the stable release. Thanks, again!

Curtis Hovey (sinzui)
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.