StaticIPAddress matching query does not exist.

Bug #2015411 reported by Chris Johnston
40
This bug affects 7 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Committed
High
Anton Troyanov
3.3
Fix Committed
High
Anton Troyanov
3.4
In Progress
High
Anton Troyanov

Bug Description

I am deploying MAAS using FCE. During one of the FCE steps it is configuring the nodes. The first time that I configure the nodes things work correctly. If I realize that I need to change something and then reconfigure the nodes I run into this error when it runs the command `maas <user> machine restore-networking-configuration <system_id>`. I am seeing this with 3.3/edge, but if I deploy with 3.2 I don't see the same error.

2023-04-05 20:15:21 maasserver: [error] ################################ Exception: StaticIPAddress matching query does not exist. ################################
2023-04-05 20:15:21 maasserver: [error] Traceback (most recent call last):
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__
    rel_obj = self.field.get_cached_value(instance)
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
    return instance._state.fields_cache[cache_name]
KeyError: 'gateway_link_ipv4'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/snap/maas/27109/lib/python3.10/site-packages/maasserver/utils/views.py", line 293, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/snap/maas/27109/lib/python3.10/site-packages/maasserver/api/support.py", line 62, in __call__
    response = super().__call__(request, *args, **kwargs)
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func
    response = func(*args, **kwargs)
  File "/snap/maas/27109/usr/lib/python3.10/dist-packages/piston3/resource.py", line 238, in __call__
    stream = srl.render(request)
  File "/snap/maas/27109/usr/lib/python3.10/dist-packages/piston3/emitters.py", line 448, in render
    self.construct(), cls=DjangoJSONEncoder, ensure_ascii=False, indent=4
  File "/snap/maas/27109/usr/lib/python3.10/dist-packages/piston3/emitters.py", line 352, in construct
    return _any(self.data, self.fields)
  File "/snap/maas/27109/usr/lib/python3.10/dist-packages/piston3/emitters.py", line 122, in _any
    ret = _model(thing, fields)
  File "/snap/maas/27109/usr/lib/python3.10/dist-packages/piston3/emitters.py", line 269, in _model
    ret[maybe_field] = _any(met_fields[maybe_field](data))
  File "/snap/maas/27109/lib/python3.10/site-packages/maasserver/api/machines.py", line 564, in default_gateways
    gateways = machine.get_default_gateways()
  File "/snap/maas/27109/lib/python3.10/site-packages/maasserver/models/node.py", line 5133, in get_default_gateways
    if self.gateway_link_ipv4 is not None:
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
    rel_obj = self.get_object(instance)
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
    return qs.get(self.field.get_reverse_related_filter(instance))
  File "/snap/maas/27109/usr/lib/python3/dist-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
maasserver.models.staticipaddress.StaticIPAddress.DoesNotExist: StaticIPAddress matching query does not exist.

I found this commit which looks like it could possibly be related:

https://git.launchpad.net/maas/commit/?id=3ac7d21924bb25c0181320c95bc52f787920fa5b

Related branches

Revision history for this message
Andre Ruiz (andre-ruiz) wrote :
Download full text (6.6 KiB)

This is happening on 3.3.2 during an Openstack deployment.

I reduced my entire bucektsconfig to 1 line, and I still get this error, even when cleaning all machines and starting from scratch without any prior config:

disks: []
nics:
  - { id: ens1f0, name: ens1f0, nic: 4, subnets: [], type: physical, fabric: openstack, vlan_id: 0, mtu: 9000 }

This is enough to give me:

ubuntu@brtlvmrs0763co:~/2023-01-30-Telefonica-OCS5-PedroGusso-OP-237579$ fce --debug build --layer maas --steps maas:configure_nodes
2023-04-12-15:16:27 root DEBUG fce --debug build --layer maas --steps maas:configure_nodes
2023-04-12-15:16:27 root DEBUG FCE version: 2.18.2
2023-04-12-15:16:27 root DEBUG [localhost]: juju export-bundle -m foundations-maas:lma-maas
2023-04-12-15:16:27 root ERROR [localhost] Command failed: juju export-bundle -m foundations-maas:lma-maas
2023-04-12-15:16:27 root ERROR [localhost] STDOUT follows:
b''
2023-04-12-15:16:27 root ERROR [localhost] STDERR follows:
ERROR controller foundations-maas not found

2023-04-12-15:16:27 root DEBUG [localhost]: juju export-bundle -m foundations-maas:openstack
2023-04-12-15:16:27 root ERROR [localhost] Command failed: juju export-bundle -m foundations-maas:openstack
2023-04-12-15:16:27 root ERROR [localhost] STDOUT follows:
b''
2023-04-12-15:16:27 root ERROR [localhost] STDERR follows:
ERROR controller foundations-maas not found

2023-04-12-15:16:27 root DEBUG Running 'zone' project check
2023-04-12-15:16:27 fce.build INFO Started building layer: maas
2023-04-12-15:16:30 fce.maas INFO Starting step: maas:configure_nodes
2023-04-12-15:16:30 root DEBUG [localhost]: maas root machines read hostname=brtlvmrs0754co domain=maas
2023-04-12-15:16:33 root DEBUG [localhost]: maas root tags create name=control-nodes
2023-04-12-15:16:35 root ERROR [localhost] Command failed: maas root tags create name=control-nodes
2023-04-12-15:16:35 root ERROR [localhost] STDOUT follows:
{"name": ["Tag with this Name already exists."]}
2023-04-12-15:16:35 root ERROR [localhost] STDERR follows:
b''
2023-04-12-15:16:35 root DEBUG [localhost]: maas root tag update-nodes control-nodes add=aws7he
2023-04-12-15:16:36 root INFO Applying config 'control-nodes' to 'brtlvmrs0754co.maas'
2023-04-12-15:16:36 root DEBUG [localhost]: maas root machines read hostname=brtlvmrs0754co domain=maas
2023-04-12-15:16:39 root INFO Skipping storage configuration.
2023-04-12-15:16:39 root INFO Setting up networking on 'aws7he'
2023-04-12-15:16:39 root INFO Clearing previous network configuration
2023-04-12-15:16:39 root DEBUG [localhost]: maas root machine restore-networking-configuration aws7he
2023-04-12-15:16:43 root ERROR [localhost] Command failed: maas root machine restore-networking-configuration aws7he
2023-04-12-15:16:43 root ERROR [localhost] STDOUT follows:
StaticIPAddress matching query does not exist.
2023-04-12-15:16:43 root ERROR [localhost] STDERR follows:
b''
2023-04-12-15:16:43 root ERROR ('brtlvmrs0754co.maas', 'control-nodes'):Command '['maas', 'root', 'machine', 'restore-networking-configuration', 'aws7he']' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/usr/local/bin/fce", line 33, in <module>
...

Read more...

Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

Subscribed field-critical on this one since deployment just got blocked.

Revision history for this message
Igor Brovtsin (igor-brovtsin) wrote (last edit ):

For some reason, Django attempts to fetch the record by the foreign key, even though it is nullable. Probably because there *was* a value in the first place, although I cannot seem to reproduce that exact failure locally. A hotfix would be to handle the error in `default_gateways`.

Changed in maas:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Igor Brovtsin (igor-brovtsin)
milestone: none → 3.4.0
Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

Removing field-critical since I have a workaround (re-install with maas 3.2 for now, upgrade again later).

Alberto Donato (ack)
Changed in maas:
milestone: 3.4.0 → 3.4.x
Revision history for this message
Navdeep (navdeep-bjn) wrote :

I just moved to MaaS 3.3 on Support's request and now facing the same issue. The issue for me happens when I have static IP's assigned to the node and run maas:configure_nodes steps. The command it fails on maas', 'admin', 'machine', 'restore-networking-configuration'<node-id>. If I go into the maas UI and delete the statically assigned IP then run the step again it works..

Revision history for this message
Vern Hart (vern) wrote :

I've encountered this on MAAS 3.3.4.

When I configure nodes (with fce) and it executes the restore-networking-configuration sub-command, I get the StaticIPAddress error. I can recreate it at the command-line:

  $ maas root machine restore-networking-configuration fk7ncm
  StaticIPAddress matching query does not exist.

If I recommission that machine and re-run the restore-networking-configuration sub-command, the error goes away but seems to pop up on a different node. I've gone through a few cycles of this whack-a-mole.

I'm not using any static IP addresses.

I have 42 machines. I just recommissioned them all at once and, when done ran the following and had positive results for every machine:

  $ for id in $(maas root machines read | jq -r '.[]|.system_id'); do
      maas root machine restore-networking-configuration $id | grep fqdn;
    done

Then, after running `fce build --layer maas --steps generate_buckets,configure_nodes` I saw no errors.

Why would this give different results than recommissioning individual nodes at a time? Maybe I got lucky?

Revision history for this message
Vern Hart (vern) wrote :

A less drastic work-around than recommissioning all nodes (or rolling back to 3.2) is the following db update:

  sudo su - maas -c "psql maasdb -c 'update maasserver_node set gateway_link_ipv4_id=null where system_id='\''cpg88q'\'';' "

Where cpg88q is the system id of the node where the restore-networking-configuration command fails.

Revision history for this message
Nobuto Murata (nobuto) wrote :

Subscribing ~field-high.

Basically `maas <PROFILE> machine restore-networking-configuration` is not working. And there is no known workaround unless doing a database surgery or completely removing and re-adding a machine.

Changed in maas:
assignee: Igor Brovtsin (igor-brovtsin) → nobody
Changed in maas:
assignee: nobody → Anton Troyanov (troyanov)
Changed in maas:
milestone: 3.4.x → 3.5.0
Changed in maas:
status: Triaged → In Progress
Changed in maas:
status: In Progress → Fix Committed
Revision history for this message
Chris Johnston (cjohnston) wrote :

This should be available in 3.3/edge: 3.3.5-13208-g.beee034cf for anyone who needs it.

Revision history for this message
Bartosz Woronicz (mastier1) wrote :

Thanks Chris.

So far I checked also re-commisioning the nodes also help, so that at least cuts the time on enlisting nodes. Looking forward to that change landing in stable.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.