Uploading a new custom image does not remove the old image
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| MAAS |
Fix Released
|
High
|
Lee Trager | ||
| 2.3 |
Triaged
|
Medium
|
Lee Trager | ||
| 2.4 |
Triaged
|
Medium
|
Lee Trager | ||
Bug Description
For context, see the thread here:
https:/
Summarizing:
Doing a "maas-region db_vacuum_lobjects” doesn’t seem to clean out the overwritten image it seems like. Below I’m doing a du -hs of /var/lib/postgresql before and after an image upload as well as after the vacuum.
13G /var/lib/postgresql
Uploading
19G /var/lib/postgresql
Cleanup database
vacuumdb: vacuuming database "maasdb"
Database vacuumed successfully.
19G /var/lib/postgresql
It seems though that the database is not shrinking, while iterating and making a new windows image over and over again, uploading it with:
maas admin boot-resources create name=windows/
Running `vacuumlo`, followed by a `VACUUM FULL` and a `REINDEX TABLE` also had no effect.
maasdb=# select id, sha256, total_size, content from maasserver_
id | sha256 | total_size | content
----+--
3 | c982d550cc03da0
6 | db99dbac5ffdd56
8 | 233f693406cce45
9 | 6ea59948ad694f2
10 | 12c895ac0ec41d1
11 | 3ff850786b9514c
12 | 8d4e6c2b4188baa
13 | 556a3192d070a40
14 | 8b1fda76f7ad259
15 | cde4daf9f6ba691
16 | 576bed34e75a61e
17 | 9002e787a4e5f3d
18 | 4b7e166571168a6
19 | e92149d78744623
20 | 63bf66548c5e6a2
21 | 06eb302764703c8
22 | 714a6903bab25c3
23 | 4d5f762ff60b58e
24 | 7a517586b2f9798
25 | b7313ab3a0a06bd
(20 rows)
maasdb=# select largefile_id, filename, filetype, extra from maasserver_
largefile_id | filename | filetype | extra
-------
3 | boot-kernel | boot-kernel | {"kpackage": "linux-generic"}
3 | boot-kernel | boot-kernel | {"kpackage": "linux-generic"}
6 | boot-kernel | boot-kernel | {"kpackage": "linux-lowlatency"}
6 | boot-kernel | boot-kernel | {"kpackage": "linux-lowlatency"}
8 | shim-signed.tar.xz | archive.tar.xz | {"src_package": "shim-signed", "src_release": "xenial", "src_version": "1.19~16.
9 | grub2-signed.tar.xz | archive.tar.xz | {"src_package": "grub2-signed", "src_release": "xenial", "src_version": "1.66.7+
10 | grub2.tar.xz | archive.tar.xz | {"src_package": "grub2", "src_release": "xenial", "src_version": "2.02~beta2-
11 | grub2.tar.xz | archive.tar.xz | {"src_package": "grub2", "src_release": "xenial", "src_version": "2.02~beta2-
12 | syslinux.tar.xz | archive.tar.xz | {"src_package": "syslinux", "src_release": "xenial", "src_version": "3:6.03+
13 | root-dd | root-dd | ""
14 | root-dd | root-dd | ""
15 | root-dd | root-dd | ""
16 | squashfs | squashfs | {}
16 | squashfs | squashfs | {}
16 | squashfs | squashfs | {}
16 | squashfs | squashfs | {}
16 | squashfs | squashfs | {}
16 | squashfs | squashfs | {}
17 | boot-initrd | boot-initrd | {"kpackage": "linux-lowlatency"}
18 | boot-kernel | boot-kernel | {"kpackage": "linux-
19 | boot-initrd | boot-initrd | {"kpackage": "linux-
20 | boot-initrd | boot-initrd | {"kpackage": "linux-lowlatency"}
21 | boot-initrd | boot-initrd | {"kpackage": "linux-generic"}
22 | boot-kernel | boot-kernel | {"kpackage": "linux-
23 | boot-initrd | boot-initrd | {"kpackage": "linux-
24 | boot-initrd | boot-initrd | {"kpackage": "linux-generic"}
25 | root-dd | root-dd | ""
(27 rows)
... but from the above, the cause seems to be that overwriting a custom image does not remove the previous boot resource file, or large object. So that could be why the vacuuming isn't working.
Related branches
- Blake Rouse (community): Approve
-
Diff: 116 lines (+53/-2)2 files modifiedsrc/maasserver/forms/__init__.py (+11/-0)
src/maasserver/forms/tests/test_bootresource.py (+42/-2)
- Mike Pontillo (community): Approve
- Newell Jensen (community): Approve
-
Diff: 532 lines (+344/-12)4 files modifiedsrc/maasserver/static/js/angular/controllers/pod_details.js (+75/-7)
src/maasserver/static/js/angular/controllers/tests/test_pod_details.js (+138/-5)
src/maasserver/static/partials/pod-details.html (+87/-0)
src/maasserver/static/scss/_tables.scss (+44/-0)
| Changed in maas: | |
| milestone: | none → 2.5.x |
| tags: | added: track |
| Tim Chen (timchen119) wrote | #1 |
| Changed in maas: | |
| milestone: | 2.5.x → 2.5.0beta2 |
| Changed in maas: | |
| assignee: | nobody → Lee Trager (ltrager) |
| tags: | added: sprint |
| Changed in maas: | |
| status: | Triaged → In Progress |
| Paul Larson (pwlars) wrote | #2 |
| Changed in maas: | |
| status: | In Progress → Fix Committed |
| Lee Trager (ltrager) wrote | #3 |
| Changed in maas: | |
| status: | Fix Committed → Fix Released |
One important symptom found here:
- Upload and importing custom images to MAAS server with same filename with different SHA256 will continue grow the `maasserver_
so I feel it's more a MAAS bug than postgres.
Current workaround is to remove the last uploaded image whenever we upload a custom image, thus the entries in `maasserver_
and after a while the postgres auto vacuum and vacuumlo will be triggered from MAAS and release the allocated space to the system.
Following MAAS CLI command will get the MAAS image list (with logged user `admin`):
$ maas admin boot-resources read
And we will get a image entry id , for example following `pc-carson-
....
{
"id": 32,
"type": "Uploaded",
"name": "pc-carson-
"title": "pc-carson-
}
Then delete the image,
$ maas admin boot-resource delete 32
Wait a while and the image(s) will be released automatically by MAAS (and postgres) and the disk space will return.