Surely. Here are the RegionD logs: ################ Preliminary information: ens18 is the interface to which REGIOND001_ADDRESS is bound. RegionDs only have ONE interface (ens18) ################ MAAS RegionD logs: 2023-07-05 13:10:37 regiond: [info] 127.0.0.1 POST /MAAS/metadata/2012-03-01/ HTTP/1.1 --> 500 INTERNAL_SERVER_ERROR (referrer: -; agent: Python-urllib/3.10) 2023-07-05 13:10:37 provisioningserver.utils.services: [warn] Couldn't report test results: HTTP error [500] 2023-07-05 13:10:43 twisted.internet.protocol.Factory: [info] RegionServer connection established (HOST:IPv6Address(type='TCP', host='::ffff:REGIOND001_ADDRESS', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:RACKD001_ADDRESS', port=45342, flowInfo=0, scopeID=0)) 2023-07-05 13:10:43 twisted.internet.protocol.Factory: [info] RegionServer connection established (HOST:IPv6Address(type='TCP', host='::ffff:REGIOND001_ADDRESS', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:RACKD001_ADDRESS', port=45356, flowInfo=0, scopeID=0)) 2023-07-05 13:10:44 maasserver.rpc.regionservice: [info] Rack controller authenticated from '::ffff:RACKD001_ADDRESS:45342'. 2023-07-05 13:10:44 maasserver.rpc.regionservice: [info] Rack controller authenticated from '::ffff:RACKD001_ADDRESS:45356'. 2023-07-05 13:10:44 maasserver.ipc: [info] Worker pid:119857 registered RPC connection to ('gqhhy7', 'REGIOND001_ADDRESS', 5252). 2023-07-05 13:10:44 maasserver.ipc: [info] Worker pid:119857 registered RPC connection to ('gqhhy7', 'REGIOND001_ADDRESS', 5252). 2023-07-05 13:10:50 maasserver: [error] ################################ Exception: get() returned more than one ScriptResult -- it returned 2! ################################ 2023-07-05 13:10:50 maasserver: [error] Traceback (most recent call last): File "/snap/maas/28521/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/utils/views.py", line 293, in view_atomic_with_post_commit_savepoint return view_atomic(*args, **kwargs) File "/usr/lib/python3.10/contextlib.py", line 79, in inner return func(*args, **kwds) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/api/support.py", line 62, in __call__ response = super().__call__(request, *args, **kwargs) File "/snap/maas/28521/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func response = func(*args, **kwargs) File "/snap/maas/28521/usr/lib/python3.10/dist-packages/piston3/resource.py", line 197, in __call__ result = self.error_handler(e, request, meth, em_format) File "/snap/maas/28521/usr/lib/python3.10/dist-packages/piston3/resource.py", line 195, in __call__ result = meth(request, *args, **kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/api/support.py", line 370, in dispatch return function(self, request, *args, **kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 860, in signal target_status = process(node, request, status) File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 682, in _process_commissioning self._store_results( File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 559, in _store_results process_file( File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 292, in process_file script_result, _ = script_set.scriptresult_set.get_or_create( File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 683, in get_or_create return super(RelatedManager, self.db_manager(db)).get_or_create(**kwargs) File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/query.py", line 581, in get_or_create return self.get(**kwargs), False File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/query.py", line 439, in get raise self.model.MultipleObjectsReturned( metadataserver.models.scriptresult.ScriptResult.MultipleObjectsReturned: get() returned more than one ScriptResult -- it returned 2! 2023-07-05 13:11:01 maasserver: [error] Error while calling ScanNetworks: Unable to get RPC connection for rack controller 'maas-rackd001' (gqhhy7). 2023-07-05 13:11:01 maasserver: [error] Error while calling ScanNetworks: Unable to get RPC connection for rack controller 'maas-rackd002' (gbeh6f). 2023-07-05 13:11:01 maasserver: [error] Error while calling ScanNetworks: Unable to get RPC connection for rack controller 'maas-rackd003' (w4ffg4). 2023-07-05 13:11:01 maasserver.regiondservices.active_discovery: [info] Active network discovery: Unable to initiate network scanning on any rack controller. Verify that the rack controllers are started and have connected to the region. 2023-07-05 13:11:01 twisted.internet.defer: [critical] Unhandled error in Deferred: 2023-07-05 13:11:01 twisted.internet.defer: [critical] Traceback (most recent call last): Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\x04\x00\t', 98, 'Address already in use') 2023-07-05 13:11:04 maasserver: [error] ################################ Exception: get() returned more than one ScriptResult -- it returned 2! ################################ 2023-07-05 13:11:04 maasserver: [error] Traceback (most recent call last): File "/snap/maas/28521/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/utils/views.py", line 293, in view_atomic_with_post_commit_savepoint return view_atomic(*args, **kwargs) File "/usr/lib/python3.10/contextlib.py", line 79, in inner return func(*args, **kwds) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/api/support.py", line 62, in __call__ response = super().__call__(request, *args, **kwargs) File "/snap/maas/28521/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func response = func(*args, **kwargs) File "/snap/maas/28521/usr/lib/python3.10/dist-packages/piston3/resource.py", line 197, in __call__ result = self.error_handler(e, request, meth, em_format) File "/snap/maas/28521/usr/lib/python3.10/dist-packages/piston3/resource.py", line 195, in __call__ result = meth(request, *args, **kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/api/support.py", line 370, in dispatch return function(self, request, *args, **kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 860, in signal target_status = process(node, request, status) File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 682, in _process_commissioning self._store_results( File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 559, in _store_results process_file( File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 292, in process_file script_result, _ = script_set.scriptresult_set.get_or_create( File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 683, in get_or_create return super(RelatedManager, self.db_manager(db)).get_or_create(**kwargs) File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/query.py", line 581, in get_or_create return self.get(**kwargs), False File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/query.py", line 439, in get raise self.model.MultipleObjectsReturned( metadataserver.models.scriptresult.ScriptResult.MultipleObjectsReturned: get() returned more than one ScriptResult -- it returned 2! 2023-07-05 13:11:07 maasserver: [error] ################################ Exception: ################################ 2023-07-05 13:11:07 maasserver: [error] Traceback (most recent call last): File "/snap/maas/28521/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/utils/views.py", line 293, in view_atomic_with_post_commit_savepoint return view_atomic(*args, **kwargs) File "/usr/lib/python3.10/contextlib.py", line 79, in inner return func(*args, **kwds) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/api/support.py", line 62, in __call__ response = super().__call__(request, *args, **kwargs) File "/snap/maas/28521/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func response = func(*args, **kwargs) File "/snap/maas/28521/usr/lib/python3.10/dist-packages/piston3/resource.py", line 197, in __call__ result = self.error_handler(e, request, meth, em_format) File "/snap/maas/28521/usr/lib/python3.10/dist-packages/piston3/resource.py", line 195, in __call__ result = meth(request, *args, **kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/api/support.py", line 370, in dispatch return function(self, request, *args, **kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 860, in signal target_status = process(node, request, status) File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 682, in _process_commissioning self._store_results( File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 565, in _store_results script_result.store_result( File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/models/scriptresult.py", line 270, in store_result assert self.status in SCRIPT_STATUS_RUNNING_OR_PENDING AssertionError 2023-07-05 13:11:07 regiond: [info] 127.0.0.1 POST /MAAS/metadata/2012-03-01/ HTTP/1.1 --> 500 INTERNAL_SERVER_ERROR (referrer: -; agent: Python-urllib/3.10) 2023-07-05 13:11:07 provisioningserver.utils.services: [warn] Couldn't report test results: HTTP error [500] 2023-07-05 13:11:20 maasserver: [error] ################################ Exception: get() returned more than one ScriptResult -- it returned 2! ################################ 2023-07-05 13:11:20 maasserver: [error] Traceback (most recent call last): File "/snap/maas/28521/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/utils/views.py", line 293, in view_atomic_with_post_commit_savepoint return view_atomic(*args, **kwargs) File "/usr/lib/python3.10/contextlib.py", line 79, in inner return func(*args, **kwds) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/api/support.py", line 62, in __call__ response = super().__call__(request, *args, **kwargs) File "/snap/maas/28521/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func response = func(*args, **kwargs) File "/snap/maas/28521/usr/lib/python3.10/dist-packages/piston3/resource.py", line 197, in __call__ result = self.error_handler(e, request, meth, em_format) File "/snap/maas/28521/usr/lib/python3.10/dist-packages/piston3/resource.py", line 195, in __call__ result = meth(request, *args, **kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/maasserver/api/support.py", line 370, in dispatch return function(self, request, *args, **kwargs) File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 860, in signal target_status = process(node, request, status) File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 682, in _process_commissioning self._store_results( File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 559, in _store_results process_file( File "/snap/maas/28521/lib/python3.10/site-packages/metadataserver/api.py", line 292, in process_file script_result, _ = script_set.scriptresult_set.get_or_create( File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/fields/related_descriptors.py", line 683, in get_or_create return super(RelatedManager, self.db_manager(db)).get_or_create(**kwargs) File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/query.py", line 581, in get_or_create return self.get(**kwargs), False File "/snap/maas/28521/usr/lib/python3/dist-packages/django/db/models/query.py", line 439, in get raise self.model.MultipleObjectsReturned( metadataserver.models.scriptresult.ScriptResult.MultipleObjectsReturned: get() returned more than one ScriptResult -- it returned 2! ################ This is the syslog: Jul 5 13:10:08 maas-regiond-ha001 systemd[1]: snap.maas.maas.1b422849-d883-48c4-b715-2c8c082a7a44.scope: Succeeded. Jul 5 13:10:14 maas-regiond-ha001 chronyd[869]: Selected source NTP_SERVER_ADDRESS Jul 5 13:10:20 maas-regiond-ha001 chronyd[869]: Can't synchronise: no majority Jul 5 13:10:31 maas-regiond-ha001 kernel: [5425968.876322] audit: type=1400 audit(1688562631.470:1769208): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sda" pid=1627306 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:10:31 maas-regiond-ha001 kernel: [5425968.889787] audit: type=1400 audit(1688562631.485:1769209): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sr0" pid=1627306 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:10:37 maas-regiond-ha001 kernel: [5425975.158911] audit: type=1400 audit(1688562637.637:1769210): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/etc/wgetrc" pid=1627324 comm="wget" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:10:37 maas-regiond-ha001 kernel: [5425975.213956] audit: type=1400 audit(1688562637.696:1769211): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sda" pid=1627327 comm="tmp.B33dw59gaq" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:10:37 maas-regiond-ha001 kernel: [5425975.216118] audit: type=1400 audit(1688562637.700:1769212): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sr0" pid=1627327 comm="tmp.B33dw59gaq" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:01 maas-regiond-ha001 kernel: [5425998.484602] audit: type=1400 audit(1688562661.454:1769213): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sda" pid=1627367 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:01 maas-regiond-ha001 kernel: [5425998.489373] audit: type=1400 audit(1688562661.458:1769214): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sr0" pid=1627367 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:01 maas-regiond-ha001 kernel: [5425998.874805] audit: type=1400 audit(1688562661.842:1769215): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sda" pid=1627380 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:01 maas-regiond-ha001 kernel: [5425998.885526] audit: type=1400 audit(1688562661.854:1769216): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sr0" pid=1627380 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:02 maas-regiond-ha001 kernel: [5425999.117281] audit: type=1400 audit(1688562662.086:1769217): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sda" pid=1627387 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:02 maas-regiond-ha001 kernel: [5425999.118050] audit: type=1400 audit(1688562662.086:1769218): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sr0" pid=1627387 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:02 maas-regiond-ha001 kernel: [5425999.257442] audit: type=1400 audit(1688562662.226:1769219): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sda" pid=1627394 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:02 maas-regiond-ha001 kernel: [5425999.258219] audit: type=1400 audit(1688562662.226:1769220): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sr0" pid=1627394 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:02 maas-regiond-ha001 kernel: [5425999.394155] audit: type=1400 audit(1688562662.362:1769221): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sda" pid=1627401 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:02 maas-regiond-ha001 kernel: [5425999.394791] audit: type=1400 audit(1688562662.362:1769222): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sr0" pid=1627401 comm="amd64" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:07 maas-regiond-ha001 kernel: [5426004.621416] audit: type=1400 audit(1688562667.590:1769223): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/etc/wgetrc" pid=1627414 comm="wget" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:07 maas-regiond-ha001 kernel: [5426004.664811] audit: type=1400 audit(1688562667.634:1769224): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sda" pid=1627417 comm="tmp.q1HqrSVHEz" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:07 maas-regiond-ha001 kernel: [5426004.665853] audit: type=1400 audit(1688562667.634:1769225): apparmor="DENIED" operation="open" profile="snap.maas.supervisor" name="/dev/sr0" pid=1627417 comm="tmp.q1HqrSVHEz" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jul 5 13:11:19 maas-regiond-ha001 chronyd[869]: Selected source NTP_SERVER_ADDRESS Jul 5 13:11:24 maas-regiond-ha001 chronyd[869]: Can't synchronise: no majority