Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\xf5\x88\x06', 98, 'Address already in use')

I believe this bug was caused because MAAS was picking up interfaces attached to bridges for the purpose of attaching VMs or containers, in addition to the bridges themselves.

That is, this should have been resolved with the following commit:

commit 842646142d42d95453f4b64c209f93b09f8aea7c
Author: Mike Pontillo <email address hidden>
Date: Fri Aug 24 21:08:12 2018 +0000

    LP: #1788780 - Ignore tunnel interfaces (such as those created by hypervisors) when gathering interfaces for MAAS management.

I'm going to set this to Incomplete unless it can be reproduced with the latest alpha release. Feel free to reopen it if you see it again, and I'll take another look.

The same issue was reproduced on MAAS 2.9.0beta5 (visible in rackd.log) running in a virtual as a standard package (not snap):

2020-10-20 18:16:32 provisioningserver.rackdservices.dhcp_probe_service: [info] Probe for external DHCP servers started on interfaces: mgmt0, docker0, bond0, cni0.
2020-10-20 18:17:12 provisioningserver.rackdservices.dhcp_probe_service: [info] External DHCP probe complete.
2020-10-20 18:21:32 twisted.internet.defer: [critical] Unhandled error in Deferred:
2020-10-20 18:21:32 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\xc0\xa8\x14\x05', 98, 'Address already in use')

I don't believe this issue should be marked as duplicate of the bug #1788780 or its resolution should be revisited.

The issue is intermittent, restarting maas-rackd "overcame" the issue (for now).

Hello,

It seems having same issue.

2021-02-01 06:34:23 twisted.internet.defer: [critical] Unhandled error in Deferred:
2021-02-01 06:34:23 twisted.internet.defer: [critical]

Traceback (most recent call last):
Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n[*\x15', 98, 'Address already in use')
2021-02-01 06:34:39 provisioningserver.utils.services: [critical] Failed to update and/or record network interface configuration: (b'UNHANDLED', 'Unknown Error [DUB1OSSKMLA003:pid=13615:cmd=UpdateInterfaces:ask=7e205]'); interfaces: {'broam': {'index': 10, 'mac_address': '68:05:ca:57:7d:f4', 'enabled': True, 'monitored': False, 'links': [{'mode': 'static', 'gateway': '10.91.42.1', 'address': '10.91.42.21/23'}], 'parents': ['bond0'], 'type': 'bridge', 'source': 'ipaddr'}, 'enp88s0f1': {'index': 7, 'mac_address': '68:05:ca:57:7d:f5', 'enabled': False, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'enp47s0f1': {'index': 5, 'mac_address': '68:05:ca:57:7d:b9', 'enabled': False, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'eno1': {'index': 2, 'mac_address': '7c:d3:0a:e5:36:c8', 'enabled': False, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'enp88s0f0': {'index': 6, 'mac_address': '68:05:ca:57:7d:f4', 'enabled': True, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'enp0s20f0u1u6': {'index': 8, 'mac_address': '7e:d3:0a:e5:36:cf', 'enabled': False, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'lxdbr0': {'index': 11, 'mac_address': '16:0f:d3:ae:76:3d', 'enabled': True, 'monitored': True, 'links': [], 'parents': [], 'type': 'bridge', 'source': 'ipaddr'}, 'bond0': {'index': 9, 'mac_address': '68:05:ca:57:7d:f4', 'enabled': True, 'monitored': True, 'links': [], 'parents': ['enp88s0f0', 'enp47s0f0'], 'type': 'bond', 'source': 'ipaddr'}, 'enp47s0f0': {'index': 4, 'mac_address': '68:05:ca:57:7d:b8', 'enabled': True, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}, 'eno2': {'index': 3, 'mac_address': '7c:d3:0a:e5:36:c9', 'enabled': False, 'monitored': False, 'links': [], 'parents': [], 'type': 'physical', 'source': 'ipaddr'}}

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1000, in updateInterfaces
    yield self._updateInterfaces(interfaces)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py"...

We are having this problem with MAAS 3.3.3 (Using 3 RackDs and 3 RegionDs, they are not running stable, as are flapping in and out of functional services.

Our RackDs have 2 interfaces each. One for the untagged net and one for the OOB-Net to reach the BMCs.

2023-05-22 08:45:38 twisted.internet.defer: [critical] Unhandled error in Deferred:
2023-05-22 08:45:38 twisted.internet.defer: [critical]
 Traceback (most recent call last):
 Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\x04\x00\x0c', 98, 'Address already in use')

Could this be a regression, potentially?
Besides, I do not think that this bug is a duplicate of https://bugs.launchpad.net/maas/+bug/1788780 as we are not using libvirt on our RackDs and still encounter the problem.

Since this bug affects a different version than ours (3.3.3) I will create a new bugreport and link to this as duplicate. I am sorry for the changes to the ticket, as I am still just learning how to use launchpad :)

Issue LP:2020798 seems related but occurs on a newer version of MAAS and has a different failure mode. This issue shows "Failed to update and/or record network interface configuration" which is not reproducible on the current MAAS.

At the least, we will improve the error message to make the root cause easier to diagnose, should this occur again.

Linking another launchpad bug with similar errors being thrown https://bugs.launchpad.net/maas/+bug/2030781

We are hitting a similar error on a customer environment after upgrading from Maas 2.9 to 3.2.

2023-09-12 06:55:19 provisioningserver.rackdservices.dhcp_probe_service: [info] Probe for external DHCP servers started on interfaces: brappformix, brext, broam.
2023-09-12 06:55:20 twisted.internet.defer: [critical] Unhandled error in Deferred:
2023-09-12 06:55:20 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'666\t', 98, 'Address already in use')

2023-09-12 06:55:20 twisted.internet.defer: [critical] Unhandled error in Deferred:
2023-09-12 06:55:20 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\x0f\xa1\x05', 98, 'Address already in use')

2023-09-12 06:55:20 twisted.internet.defer: [critical] Unhandled error in Deferred:
2023-09-12 06:55:20 twisted.internet.defer: [critical]
        Traceback (most recent call last):
        Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\x0f\xa8\n', 98, 'Address already in use')

2023-09-12 06:48:26 provisioningserver.utils.services: [critical] Failed to update and/or record network interface configuration: Expecting value: line 1 column 1 (char 0); interfaces: {'bond0': {'type': 'bond', 'mac_address': 'Redacted', 'links': [], 'enabled': True, 'parents': ['eno12409np1', 'eno12399np0'], 'source': 'machine-resources', 'monitored': True}, 'bond0.402': {'type': 'vlan', 'mac_address': 'Redacted', 'links': [], 'enabled': True, 'parents': ['bond0'], 'source': 'machine-resources', 'vid': 402, 'monitored': False}, 'bond0.403': {'type': 'vlan', 'mac_address': 'Redacted', 'links': [], 'enabled': True, 'parents': ['bond0'], 'source': 'machine-resources', 'vid': 403, 'monitored': False}, 'brappformix': {'type': 'bridge', 'mac_address': 'Redacted', 'links': [{'mode': 'static', 'address': 'Redacted'}], 'enabled': True, 'parents': ['bond0.403'], 'source': 'machine-resources', 'monitored': False}, 'brext': {'type': 'bridge', 'mac_address': 'Redacted', 'links': [{'mode': 'static', 'address': 'Redacted'}], 'enabled': True, 'parents': ['bond0.402'], 'source': 'machine-resources', 'monitored': False}, 'broam': {'type': 'bridge', 'mac_address': 'Redacted', 'links': [{'mode': 'static', 'address': 'Redacted', 'gateway': 'Redacted'}, {'mode': 'static', 'address': 'Redacted', 'gateway': 'Redacted'}], 'enabled': True, 'parents': ['bond0'], 'source': 'machine-resources', 'monitored': False}, 'eno12399np0': {'type': 'physical', 'mac_address': 'Redacted', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno12409np1': {'type': 'physical', 'mac_address': 'Redacted', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno8303': {'type': 'physical', 'mac_address': 'Redacted', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'eno8403': {'type': 'physical', 'mac_address': 'Redacted', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resourc...