MAAS

[2.4] MAAS UI becomes unresponsive (slow) when there's disconnected controllers/RPC errors

  1. Series 2.4
  2. Bug #1762461
Bug #1762461 reported by Andres Rodriguez
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Lee Trager
MAAS 2.5.0alpha1
2.4
In Progress
Critical
Lee Trager
MAAS 2.4.3

Bug Description

Upgrading my primary region/rack, MAAS bacame bvery unresponsive and in the logs I saw the following errors.

This seems like a new recurrence of this old bug: https://bugs.launchpad.net/maas/+bug/1360004

2018-04-09 15:39:05 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'maas00' (6g7yrg).
2018-04-09 15:39:06 maasserver: [error] Error while calling DescribePowerTypes: Unable to get RPC connection for rack controller 'node01' (4rcbwp).
2018-04-09 15:39:10 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'maas00' (6g7yrg).
2018-04-09 15:39:10 maasserver: [error] Error while calling DescribePowerTypes: Unable to get RPC connection for rack controller 'node01' (4rcbwp).
2018-04-09 15:39:16 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'maas00' (6g7yrg).
2018-04-09 15:39:16 maasserver: [error] Error while calling DescribePowerTypes: Unable to get RPC connection for rack controller 'node01' (4rcbwp).
2018-04-09 15:39:20 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'maas00' (6g7yrg).
2018-04-09 15:39:20 maasserver: [error] Error while calling DescribePowerTypes: Unable to get RPC connection for rack controller 'node01' (4rcbwp).

See original description

Related branches

~ltrager/maas:lp1762461
MAAS Lander: Needs Fixing
Blake Rouse (community): Approve
Diff: 2375 lines (+638/-548)
26 files modified
src/maasserver/api/tests/test_enlistment.py (+26/-29)
src/maasserver/api/tests/test_licensekey.py (+36/-27)
src/maasserver/api/tests/test_machine.py (+6/-20)
src/maasserver/api/tests/test_machines.py (+3/-12)
src/maasserver/forms/__init__.py (+3/-4)
src/maasserver/forms/settings.py (+2/-4)
src/maasserver/forms/tests/test_bootresource.py (+14/-11)
src/maasserver/forms/tests/test_deploy.py (+14/-13)
src/maasserver/forms/tests/test_licensekey.py (+39/-51)
src/maasserver/forms/tests/test_machine.py (+37/-30)
src/maasserver/models/bootresource.py (+5/-8)
src/maasserver/models/node.py (+50/-33)
src/maasserver/models/tests/test_bootresource.py (+3/-5)
src/maasserver/models/tests/test_node.py (+115/-75)
src/maasserver/testing/factory.py (+1/-1)
src/maasserver/testing/osystems.py (+41/-8)
src/maasserver/tests/test_compose_preseed.py (+0/-2)
src/maasserver/tests/test_node_action.py (+9/-14)
src/maasserver/utils/osystems.py (+77/-43)
src/maasserver/utils/tests/test_osystems.py (+77/-106)
src/maasserver/views/settings.py (+2/-1)
src/maasserver/views/tests/test_settings.py (+21/-18)
src/maasserver/views/tests/test_settings_license_keys.py (+43/-24)
src/maasserver/websockets/handlers/general.py (+3/-3)
src/maasserver/websockets/handlers/tests/test_general.py (+2/-1)
src/provisioningserver/testing/os.py (+9/-5)
Revision history for this message
Andres Rodriguez (andreserl) wrote :

This seems very similar to:

https://bugs.launchpad.net/maas/+bug/1360004

description: updated
Changed in maas:
milestone: none → 2.4.0beta2
importance: Undecided → High
status: New → Triaged
summary: - [2.4] MAAS becomes unresponsive when there's RPC errors
+ [2.4] MAAS UI becomes unresponsive when there's RPC errors
Andres Rodriguez (andreserl)
Changed in maas:
importance: High → Critical
assignee: nobody → Lee Trager (ltrager)
Revision history for this message
Andres Rodriguez (andreserl) wrote : Re: [2.4] MAAS UI becomes unresponsive when there's RPC errors

https://bugs.launchpad.net/maas/+bug/1761601 is a duplicate of this.

Revision history for this message
Andres Rodriguez (andreserl) wrote :

PLease see 1762461 for more details too.

Andres Rodriguez (andreserl)
Changed in maas:
milestone: 2.4.0beta2 → 2.4.0rc1
Andres Rodriguez (andreserl)
summary: - [2.4] MAAS UI becomes unresponsive when there's RPC errors
+ [2.4] MAAS UI becomes unresponsive (slow) when there's disconnected
+ controllers/RPC errors
Changed in maas:
assignee: Lee Trager (ltrager) → Blake Rouse (blake-rouse)
Blake Rouse (blake-rouse)
Changed in maas:
assignee: Blake Rouse (blake-rouse) → nobody
status: Triaged → Incomplete
Andres Rodriguez (andreserl)
Changed in maas:
milestone: 2.4.0rc1 → 2.4.0rc2
Revision history for this message
Mark Shuttleworth (sabdfl) wrote :

Not sure where we are tracking this, but I see it in the GMAAS too. I think I have one sick rack controller. Might it have to do with the IP addresses on which RPC services listen?

Revision history for this message
Mark Shuttleworth (sabdfl) wrote :

I see this:

2018-05-06 12:47:16 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'lapsi' (4nqs36).
2018-05-06 12:47:23 -: [critical] Unhandled error in EventualResult
 Traceback (most recent call last):
 Failure: twisted.internet.defer.CancelledError:

2018-05-06 12:47:26 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'lapsi' (4nqs36).

Andres Rodriguez (andreserl)
Changed in maas:
milestone: 2.4.0rc2 → 2.4.0rc1
status: Incomplete → Confirmed
assignee: nobody → Lee Trager (ltrager)
Andres Rodriguez (andreserl)
Changed in maas:
status: Confirmed → In Progress
milestone: 2.4.0rc1 → 2.4.0rc2
Andres Rodriguez (andreserl)
Changed in maas:
milestone: 2.4.0rc2 → 2.5.0
Andres Rodriguez (andreserl)
no longer affects: maas/trunk
Changed in maas:
milestone: 2.4.1 → 2.5.0
Lee Trager (ltrager)
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl)
Changed in maas:
milestone: 2.5.0 → 2.5.0alpha1
Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
Revision history for this message
Phil Merricks (seffyroff) wrote :

I'm getting a similar experience on 2.5.0 final here. 3 rack controllers offline and I've been waiting 45 mins to bounce the region service. This is my second attempt - the first time the box eventually ground to a complete halt.

Revision history for this message
Lee Trager (ltrager) wrote :

Hi Phil,

Could you please open a new bug and attach your MAAS logs?

Revision history for this message
Phil Merricks (seffyroff) wrote :

I will do that next time I'm running a deployment. FYI I resolved this by:

1: Restoring the racks and region to their prior state (before I shut down the racks to migrate them)
2: Deleting the racks from the region controller via the Controllers UI

With that, the Region Controller played nice and allowed me to migrate it very smoothly. Hurrah!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.