after upgrading model from 1.25 to v2.x, juju migrate fails on lxd containers

Bug #1834404 reported by Xav Paice
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
Low
Unassigned

Bug Description

Juju client 2.6.4, current model (and controller) 2.6.4 all using Trusty.

The original install was 1.2x, maas 1.9, with multiple units using LXC containers.

This particular model was upgraded to 2.x, and the containers migrated to lxd as part of that process.

When I try to migrate the model to a new controller (Xenial), I get the following:

:~/bootstack-backups$ juju migrate --debug mymodel newcontroller
00:17:21 INFO juju.cmd supercommand.go:57 running juju [2.6.4 gc go1.10.4]
00:17:21 DEBUG juju.cmd supercommand.go:58 args: []string{"/snap/juju/8241/bin/juju", "migrate", "--debug", "mymodel", "newcontroller"}
00:17:21 INFO juju.juju api.go:67 connecting to API addresses: [10.101.52.156:17070]
00:17:21 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.101.52.156:17070/api"
00:17:21 INFO juju.api apiclient.go:624 connection established to "wss://10.101.52.156:17070/api"
00:17:21 DEBUG juju.api monitor.go:35 RPC connection died
00:17:21 INFO juju.juju api.go:67 connecting to API addresses: [10.101.52.169:17070]
00:17:21 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.101.52.169:17070/api"
00:17:21 INFO juju.api apiclient.go:624 connection established to "wss://10.101.52.169:17070/api"
00:17:21 DEBUG juju.api monitor.go:35 RPC connection died
00:17:21 INFO juju.juju api.go:67 connecting to API addresses: [10.101.52.156:17070]
00:17:21 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.101.52.156:17070/api"
00:17:21 INFO juju.api apiclient.go:624 connection established to "wss://10.101.52.156:17070/api"
00:17:21 DEBUG juju.api monitor.go:35 RPC connection died
ERROR source prechecks failed: machine 0/lxd/0 not running
00:17:21 DEBUG cmd supercommand.go:496 error stack:
source prechecks failed: machine 0/lxd/0 not running
/build/juju/parts/juju/go/src/github.com/juju/juju/api/controller/controller.go:333:

The container mentioned is one of the containers that was migrated from lxc.

Digging in the db using db.statuses.find( {_id: { $regex: /0\/lxd\// }}).pretty() I see that the 'status' and 'statusinfo' fields aren't populated on containers that were migrated. Restarting the machine agent doesn't seem to affect this.

Good status:

{
        "_id" : "43759692-2e64-4550-8dd4-59d1572213e6:m#0/lxd/17#instance",
        "model-uuid" : "43759692-2e64-4550-8dd4-59d1572213e6",
        "neverset" : false,
        "status" : "running",
        "statusdata" : {

        },
        "statusinfo" : "Container started",
        "txn-queue" : [ ],
        "txn-revno" : NumberLong(52),
        "updated" : NumberLong("1556832174380495695")
}

Not so good:

{
        "_id" : "43759692-2e64-4550-8dd4-59d1572213e6:m#0/lxd/14#instance",
        "model-uuid" : "43759692-2e64-4550-8dd4-59d1572213e6",
        "neverset" : false,
        "status" : "",
        "statusdata" : {

        },
        "statusinfo" : "",
        "txn-queue" : [ ],
        "txn-revno" : NumberLong(2),
        "updated" : NumberLong("1538373101870144741")
}

juju show-machine shows something very similar:

Good:
      0/lxd/17:
        juju-status:
          current: started
          since: 26 Jun 2019 04:30:57Z
          version: 2.6.4
        dns-name: 10.101.52.166
        ip-addresses:
        - 10.101.52.166
        - 10.101.60.1
        instance-id: juju-2213e6-0-lxd-17
        machine-status:
          current: running
          message: Container started
          since: 02 May 2019 21:22:54Z
        modification-status:
          current: applied
          since: 26 Jun 2019 05:32:03Z
        series: trusty
        network-interfaces:
          eth0:
            ip-addresses:
            - 10.101.52.166
            mac-address: 00:16:3e:35:93:03
            gateway: 10.101.52.1
            dns-nameservers:
            - 10.101.52.6
            space: space-0
            is-up: true
          eth1:
            ip-addresses:
            - 10.101.60.1
            mac-address: 00:16:3e:d7:79:3e
            space: space-0
            is-up: true

Not so good:
      0/lxd/0:
        juju-status:
          current: started
          since: 26 Jun 2019 04:30:05Z
          version: 2.6.4
        dns-name: 10.101.52.90
        ip-addresses:
        - 10.101.52.90
        instance-id: juju-2213e6-0-lxd-0
        machine-status:
          since: 01 Oct 2018 05:51:41Z
        modification-status:
          current: applied
          since: 26 Jun 2019 05:32:03Z
        series: trusty
        hardware: arch=amd64

      0/lxd/14:
        juju-status:
          current: started
          since: 26 Jun 2019 04:30:07Z
          version: 2.6.4
        dns-name: 10.101.52.96
        ip-addresses:
        - 10.101.52.241
        - 10.101.52.96
        instance-id: juju-2213e6-0-lxd-14
        machine-status:
          since: 01 Oct 2018 05:51:41Z
        modification-status:
          current: applied
          since: 26 Jun 2019 05:32:03Z
        series: trusty
        hardware: arch=amd64

(note also the lack of network-interfaces)

Revision history for this message
Tim Penhey (thumper) wrote :

We need to find out why the machine agent on the lxd machines aren't updating the link-layer devices or the provider status.

Changed in juju:
status: New → Triaged
importance: Undecided → Medium
tags: added: lxd model-migration
Revision history for this message
Canonical Juju QA Bot (juju-qa-bot) wrote :

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.

Changed in juju:
importance: Medium → Low
tags: added: expirebugs-bot
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.