MAAS

Cannot delete Virsh Pod

Bug #1733442 reported by Newell Jensen on 2017-11-20

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	MAAS	Fix Released	Medium	Newell Jensen	MAAS 2.8.0b1

Bug Description

After creating a Virsh Pod in MAAS, I cannot delete the pod and get the follow stacktrace showing in the UI:

The stacktraces when trying to delete the Pod in regiond.log are:

==> /var/log/maas/regiond.log <==
2017-11-20 15:47:59 maasserver.websockets.protocol: [critical] Error on request (2969) pod.delete: <ProtectedError instance at 0x7f66cb73fd68 with str error:
  Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/django/db/models/fields/related.py", line 600, in __get__
     rel_obj = getattr(instance, self.cache_name)
AttributeError: 'Subnet' object has no attribute '_vlan_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/python/reflect.py", line 443, in safe_str
     return str(o)
   File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 141, in __repr__
     return repr(data)
   File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 496, in __repr__
     u = six.text_type(self)
   File "/usr/lib/python3/dist-packages/maasserver/models/subnet.py", line 434, in __str__
     self.name, self.cidr, self.vlan.vid)
   File "/usr/lib/python3/dist-packages/django/db/models/fields/related.py", line 613, in __get__
     qs = qs.filter(**params)
   File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 679, in filter
     return self._filter_or_exclude(False, *args, **kwargs)
   File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 697, in _filter_or_exclude
     clone.query.add_q(Q(*args, **kwargs))
   File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1310, in add_q
     clause, require_inner = self._add_q(where_part, self.used_aliases)
   File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1338, in _add_q
     allow_joins=allow_joins, split_subq=split_subq,
   File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1156, in build_filter
     value, lookups, used_joins = self.prepare_lookup_value(value, lookups, can_reuse, allow_joins)
   File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1021, in prepare_lookup_value
     if (connections[DEFAULT_DB_ALIAS].features.interprets_empty_strings_as_nulls and
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 836, in __getattr__
     "disabled." % threading.currentThread().name)
RuntimeError: Database connections in this thread (MainThread) are disabled.
>

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 842, in worker
    return target()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 875, in callInContext
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 232, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 754, in call_within_transaction
    return func_outside_txn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 561, in retrier
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/models/bmc.py", line 1029, in perform_deletion
    super(BMC, pod).delete()
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 895, in delete
    collector.collect([self])
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 229, in collect
    field.rel.on_delete(self, field, sub_objs, self.using)
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 18, in CASCADE
    source_attr=field.name, nullable=field.null)
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 229, in collect
    field.rel.on_delete(self, field, sub_objs, self.using)
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 18, in CASCADE
    source_attr=field.name, nullable=field.null)
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 229, in collect
    field.rel.on_delete(self, field, sub_objs, self.using)
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 28, in PROTECT
    sub_objs
django.db.models.deletion.ProtectedError: <ProtectedError instance at 0x7f66cb73fd68 with str error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/db/models/fields/related.py", line 600, in __get__
    rel_obj = getattr(instance, self.cache_name)
AttributeError: 'Subnet' object has no attribute '_vlan_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/python/reflect.py", line 443, in safe_str
    return str(o)
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 141, in __repr__
    return repr(data)
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 496, in __repr__
    u = six.text_type(self)
  File "/usr/lib/python3/dist-packages/maasserver/models/subnet.py", line 434, in __str__
    self.name, self.cidr, self.vlan.vid)
  File "/usr/lib/python3/dist-packages/django/db/models/fields/related.py", line 613, in __get__
    qs = qs.filter(**params)
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 679, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 697, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1310, in add_q
    clause, require_inner = self._add_q(where_part, self.used_aliases)
  File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1338, in _add_q
    allow_joins=allow_joins, split_subq=split_subq,
  File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1156, in build_filter
    value, lookups, used_joins = self.prepare_lookup_value(value, lookups, can_reuse, allow_joins)
  File "/usr/lib/python3/dist-packages/django/db/models/sql/query.py", line 1021, in prepare_lookup_value
    if (connections[DEFAULT_DB_ALIAS].features.interprets_empty_strings_as_nulls and
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 836, in __getattr__
    "disabled." % threading.currentThread().name)
RuntimeError: Database connections in this thread (MainThread) are disabled.
>

Related branches

~newell-jensen/maas:lp1733442

Merged into maas:master

Alberto Donato: Approve on 2019-12-12

MAAS Lander: Approve on 2019-12-12

Revision history for this message

Newell Jensen (newell-jensen) wrote on 2017-11-21:

Download full text (3.8 KiB)

On a fresh install (new db as well), I am still getting an error on delete but it is the more compact UI error of:

Performing delete failed: ("Cannot delete some instances of model 'VLAN' because they are referenced through a protected foreign key: 'PhysicalInterface.vlan'", [<PhysicalInterface: name=ens3, type=physical, mac=52:54:00:c3:99:ac, id=1>])

With regiond.log:

==> /var/log/maas/regiond.log <==
2017-11-20 22:25:44 maasserver.websockets.protocol: [critical] Error on request (248) pod.delete: ("Cannot delete some instances of model 'VLAN' because they are referenced through a protected foreign key: 'PhysicalInterface.vlan'", [<PhysicalInterface: name=ens3, type=physical, mac=52:54:00:c3:99:ac, id=1>])

On a fresh install (new db as well), I am still getting an error on delete but it is the more compact UI error of:

With regiond.log:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 842, in worker
    return target()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 875, in callInContext
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 232, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 754, in call_within_transaction
    return func_outside_txn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 561, in retrier
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/models/bmc.py", line 1029, in perform_deletion
    super(BMC, pod).delete()
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 895, in delete
    collector.collect([self])
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 229, in collect
    field.rel.on_delete(self, field, sub_objs, self.using)
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 18, in CASCADE
    source_attr=field.name, nullable=field.null)
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 229, in collect
    field.rel.on_delete(self, field, sub_objs, self.using)
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 18, in CASCADE
    source_attr=field.name, nullable=field.null)
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 229, in collect
    field.rel.on_delete(self, field, sub_objs, self.using)
  File "/usr/lib/python3/dist-packages/django/db/models/deletion.py", line 28, in PROTECT
    sub_objs
django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'VLAN' because they are referenced through a protected foreign key: 'PhysicalInterface.vlan'", [<PhysicalInterface: name=ens3, type=physical, mac=52:54:00:c3:99:ac, id=1>])
2017-11-20 22:26:06 regiond: [info] ::1 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)

Revision history for this message

Newell Jensen (newell-jensen) wrote on 2017-11-21:

My MAAS server was running on a Virsh VM and when using deb packages or the MAAS snap, I am able to delete the Virsh Pod without any issues. Setting to won't fix at the moment (it may come up again in the future at some point but this seems to be a rather rare issue as it seems no one else has run into it and could be an issue with my testing setup).

Changed in maas:
status:	In Progress → Invalid
status:	Invalid → Won't Fix

Andres Rodriguez (andreserl) on 2018-04-23

Changed in maas:
milestone:	2.4.x → none

Alberto Donato (ack) on 2019-11-22

Changed in maas:
importance:	Critical → Medium

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2019-11-22:

Download full text (3.3 KiB)

I found a way to consistently reproduce this issue with MAAS 2.6.2 (7841-ga10625be3-0ubuntu1~18.04.1) on KVM. Basically this happens when:
- you install a MAAS region controller on a KVM machine,
- you create a virsh pod whose address points to your host running KVM.
At this point you have a nice setup where you can simply create new VMs by using "Compose machine" from MAAS' WebUI but you can't delete the pod anymore without consistently getting the exception mentioned above (AttributeError: 'Subnet' object has no attribute '_vlan_cache').

Detailed steps to reproduce:

On your host:

```
# Install KVM:
sudo apt install virt-manager
sudo reboot # see https://askubuntu.com/q/345218/113686
virt-manager
```

In `virt-manager`:
- Create a new virtual network:
    - Name: maasisotest
    - Network: 10.17.17.0/24
    - No DHCP
    - Forwarding to physical network:
        - Destination: Any physical device
        - Mode: NAT
- Create a new VM:
    - Local install media (latest Ubuntu Server 18.04 ISO image)
    - Name: 0-maas-controller0
    - Network selection: maasisotest
    - Ubuntu installation:
        - F4 Modes: Install a minimal virtual machine
        - Install MAAS Region Controller
        - Configure network manually:
            - IP address: 10.17.17.200
            - Netmask: 255.255.255.0
            - Gateway: 10.17.17.1
            - Name server addresses: 10.17.17.1
        - Hostname: 0-maas-controller0
        - Domain name: (none)

Access MAAS’ WebUI at http://10.17.17.200:5240/MAAS :
- Finalize MAAS’ setup by accepting the default options.
- Enable DHCP:
    - Navigate to Subnets / untagged / Take action / Provide DHCP:
        - Dynamic range start IP: 10.17.17.100
        - Dynamic range end IP: 10.17.17.199

We now want MAAS to be able to create and power VMs on and off by talking to the host’s libvirt:

```
ssh 10.17.17.200
sudo apt install ssh

# Upgrade from MAAS 2.4 to 2.6:
sudo add-apt-repository ppa:maas/2.6
sudo apt update
sudo apt upgrade

# Enable the `maas` user and switch to it:
sudo chsh -s /bin/bash maas
sudo su - maas

# Generate a keypair and get it trusted by your host:
ssh-keygen -N ‘’
ssh-copy-id -i ~/.ssh/id_rsa.pub myuser@10.17.17.1

# Check that MAAS can now communicate with the host’s libvirt without
# having to type a password:
virsh -c qemu+ssh://myuser@10.17.17.1/system list --all
  => Id Name State
  => -----------------------------------
  => 1 0-maas-controller0 running
exit
exit
```

Disable DHCP on libvirt’s `default` network by typing this directly on the host:

```
# Stop the `default` network:
virsh net-destroy default

# At this point we could remove DHCP from the network’s config and restart
# it but it’s easier to just keep it stopped.
# virsh net-edit default # remove <dhcp></dhcp>
# virsh net-start default

virsh net-autostart --disable default
virsh net-list --all
```

From MAAS’ WebUI:
- Create a Virsh pod:
    - Navigate to Pods / Add pod:
        - Pod name: mypod
        - Pod type: Virsh (virtual systems)
        - Virsh address: qemu+ssh://myuser@10.17.17.1/system
        - Virsh password: (none)
    - Do Pods / mypod / Take action / Refresh
    - Do P...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Changed in maas:
status:	Triaged → In Progress
milestone:	none → next

Changed in maas:
milestone:	next → 2.8.0b1
status:	Fix Committed → Fix Released