[2.0rc2] RackController.get_image_sync_status causes huge load on regiond process
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
Critical
|
Blake Rouse | ||
2.0 |
Fix Released
|
Critical
|
Blake Rouse |
Bug Description
The call in the handlers/
ControllerHandl
node.get_
BootResource.
The reason it is slow is the way MAAS calculates the size of the each largefile in the database. It does this by opening each largeobject seeking to the end and getting the size. This was fine in 1.9 when the cluster page was not polled like it is today in MAAS 2.0.
@property
def size(self):
"""Size of content."""
with self.content.
size = stream.tell()
return size
This needs to be converted to a field in the database holding the current size of the largefile instead of calculating in from the largeobject. This will speed up the BootResource.
This is currently affecting OIL, to alleviate the issue the following was performed:
class RackController(
...blah...
def get_image_
return 'unknown'
... the original function...
Related branches
- Gavin Panella (community): Approve
-
Diff: 348 lines (+97/-37)13 files modifiedsrc/maasserver/api/boot_resources.py (+2/-0)
src/maasserver/bootresources.py (+8/-1)
src/maasserver/migrations/builtin/maasserver/0067_add_size_to_largefile.py (+36/-0)
src/maasserver/migrations/builtin/maasserver/0068_drop_node_system_id_sequence.py (+1/-1)
src/maasserver/migrations/builtin/maasserver/0069_add_previous_node_status_to_node.py (+1/-1)
src/maasserver/models/bootresource.py (+9/-2)
src/maasserver/models/bootresourceset.py (+23/-13)
src/maasserver/models/largefile.py (+4/-10)
src/maasserver/models/tests/test_bootresourceset.py (+2/-0)
src/maasserver/models/tests/test_largefile.py (+2/-7)
src/maasserver/static/js/angular/directives/controller_image_status.js (+4/-1)
src/maasserver/testing/factory.py (+2/-1)
src/maasserver/tests/test_bootresources.py (+3/-0)
- Blake Rouse (community): Approve
-
Diff: 322 lines (+95/-35)11 files modifiedsrc/maasserver/api/boot_resources.py (+2/-0)
src/maasserver/bootresources.py (+8/-1)
src/maasserver/migrations/builtin/maasserver/0067_add_size_to_largefile.py (+36/-0)
src/maasserver/models/bootresource.py (+9/-2)
src/maasserver/models/bootresourceset.py (+23/-13)
src/maasserver/models/largefile.py (+4/-10)
src/maasserver/models/tests/test_bootresourceset.py (+2/-0)
src/maasserver/models/tests/test_largefile.py (+2/-7)
src/maasserver/static/js/angular/directives/controller_image_status.js (+4/-1)
src/maasserver/testing/factory.py (+2/-1)
src/maasserver/tests/test_bootresources.py (+3/-0)
- Andres Rodriguez (community): Approve
-
Diff: 90 lines (+54/-6)2 files modifieddebian/changelog (+44/-0)
debian/maas-region-controller.postinst (+10/-6)
tags: | added: oil |
Changed in maas: | |
assignee: | nobody → Blake Rouse (blake-rouse) |
status: | Triaged → In Progress |
milestone: | 2.0.0 → 2.1.0 |
Changed in maas: | |
status: | In Progress → Fix Committed |
Changed in maas: | |
status: | Fix Committed → Fix Released |
milestone: | 2.0.1 → none |
tags: | added: auto-sanity tpe-lab |
tags: |
added: taipei-lab removed: tpe-lab |
Adding the top output showing the load:
ubuntu@ maas2-productio n:~$ top
top - 22:09:40 up 1 day, 17:51, 1 user, load average: 5.61, 5.84, 5.63
Tasks: 71 total, 2 running, 69 sleeping, 0 stopped, 0 zombie
%Cpu(s): 38.2 us, 5.4 sy, 0.0 ni, 54.6 id, 0.0 wa, 0.0 hi, 1.8 si, 0.0 st
KiB Mem : 16300676 total, 13767796 free, 1751532 used, 781348 buff/cache
KiB Swap: 16645628 total, 15951660 free, 693968 used. 13767796 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21436 maas 20 0 1221024 249004 8400 S 111.3 1.5 3813:45 twistd3
21438 maas 20 0 1810808 262696 8556 S 111.3 1.6 4895:17 twistd3
21429 maas 20 0 1223360 245912 8384 S 107.3 1.5 3464:28 twistd3
21434 maas 20 0 1147128 247360 8512 S 70.7 1.5 1884:17 twistd3
25142 postgres 20 0 295304 21944 19888 S 19.3 0.1 0:00.58 postgres
24964 postgres 20 0 296604 102552 99740 S 11.0 0.6 0:01.97 postgres
25125 postgres 20 0 295304 21940 19888 S 8.3 0.1 0:00.85 postgres
25126 postgres 20 0 295304 21936 19884 S 8.3 0.1 0:00.84 postgres
25137 postgres 20 0 295304 21936 19888 R 8.3 0.1 0:00.73 postgres
25139 postgres 20 0 295304 21936 19884 S 8.3 0.1 0:00.50 postgres
25140 postgres 20 0 295304 18584 16584 S 8.3 0.1 0:00.43 postgres
25154 postgres 20 0 295304 18588 16592 S 6.3 0.1 0:00.19 postgres