Juju creates extra/redundant VMs when using MAAS vmhost as backing cloud

Bug #2039210 reported by Dagmawi Biru

This bug report was marked for expiration 185 days ago. (find out why)

14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Incomplete
Undecided
Unassigned
MAAS
Incomplete
Undecided
Unassigned

Bug Description

Some weird race condition between Juju's request towards MAAS causes multiple duplicate
VM hosts to be created when using MAAS as the backing cloud with a either KVM or LXC VM vhost configured.

Steps to reproduce:
------
1. Deploy MAAS 3.x
2. Add a LXC VM host/hypervisor to MAAS so that it will be used to spin up VMs
3. Bootstrap Juju 2.x/stable controller using MAAS as the backing cloud
4. Deploy multiple VMs using the LXC VM host, for example via:
    $ juju add-machine -n3

Expected result

The exact count of machines desired by the model or -n argument are spun up.

Actual result:

Redundant/errnoeous VMs are spun up (cheif-wolf, bright-wew, super-moose):

l+-------------+---------+----------------------+------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------+---------+----------------------+------+-----------------+-----------+
| bright-ewe | STOPPED | | | VIRTUAL-MACHINE | 0 |
+-------------+---------+----------------------+------+-----------------+-----------+
| chief-wolf | STOPPED | | | VIRTUAL-MACHINE | 0 |
+-------------+---------+----------------------+------+-----------------+-----------+
| deep-goose | RUNNING | 172.16.66.195 (eth0) | | VIRTUAL-MACHINE | 0 |
+-------------+---------+----------------------+------+-----------------+-----------+
| hip-goat | RUNNING | 172.16.66.219 (eth0) | | VIRTUAL-MACHINE | 0 |
+-------------+---------+----------------------+------+-----------------+-----------+
| steady-cat | RUNNING | 172.16.66.151 (eth0) | | VIRTUAL-MACHINE | 0 |
+-------------+---------+----------------------+------+-----------------+-----------+
| super-moose | STOPPED | | | VIRTUAL-MACHINE | 0 |
+-------------+---------+----------------------+------+-----------------+-----------+
| valued-drum | RUNNING | 172.16.66.207 (eth0) | | VIRTUAL-MACHINE | 0 |
+-------------+---------+----------------------+------+-----------------+-----------+

I haven't narrowed down what exactly causes this -- is Juju not getting a reply from MAAS
fast enough and sending duplicate create requests?

Dagmawi Biru (dagbiru)
tags: added: kvm maas-provider
Revision history for this message
Dagmawi Biru (dagbiru) wrote (last edit ):

Just an addendum -- it also seems like once these extraneous instances are created,
neither Juju or MAAS are aware of the VMs in their respective databases, but these VMs still show up in lxc list output as shown in the bug description.

It almost seems like MAAS is sending LXC the go ahead to spin up the VMs, gives up on it and doesn't check with the hypervisor that LXC went ahead and created these instances (albeit in a stopped state).

Revision history for this message
Alex (borsik) wrote :

Thanks for reporting this, I can confirm same behaviour as you described.

Revision history for this message
Anton Troyanov (troyanov) wrote :

Hello Dagmawi,

It is indeed hard to say what might cause the issue.
May I ask you to attach MAAS logs (regiond and access/error logs of nginx running next to the regiond)

LXD logs might also help: lxc monitor --type=logging --pretty

Changed in maas:
status: New → Incomplete
Changed in juju:
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.