Mailman 3 Web Charm

Add MemoryMax systemd option to mailman3-web service

Bug #1991163 reported by Loïc Gomez
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mailman 3 Core Charm
Invalid
Undecided
Unassigned
Mailman 3 Web Charm
New
Undecided
Unassigned

Bug Description

Sometimes mailman3-web service uses too much memory. There's a probable memory leak, we got up to 4.5GB while it usually takes about 300MB:
https://pastebin.canonical.com/p/TtfKw978MH/ (sorry, private Canonical link).

We should add a systemd override to mailman3-web.service consisting of:
[Service]
MemoryMax=1G
Restart=on-failure

Restart=on-failure is already the default, but let's add it to make sure the service gets restarted, were it to change in the future.

For this option to work, we should also update systemd configuration /etc/systemd/system.conf (+ systemctl daemon-reload) with:
DefaultMemoryAccounting=yes

Revision history for this message
Loïc Gomez (kotodama) wrote (last edit ):

We might also want to hardfail in case it's unable to properly restart under the limit with:
[Unit]
StartLimitIntervalSec=3m
StartLimitBurst=10

Thomas suggested we could also create a mailman systemd slice and add each service into it with limits so only those resources are affected, and have the cron job to run with systemd-run or update the charm to use systemd timers.

Revision history for this message
Haw Loeung (hloeung) wrote :

Is it really the mailman3-web service that's using too much memory on these units?

What I've found is that usually it's the regular daily/weekly/monthly update jobs consuming most of the memory.

Sure, doesn't hurt to add these limits in place. FWIW, this was recently bumped from 1G to 4G so we should take that into consideration when adding these limits. Perhaps make it tunable via juju charm config options.

Changed in mailman3-core-charm:
status: New → Invalid
Revision history for this message
Loïc Gomez (kotodama) wrote :

Yes, I had a incident where mailman3-web used too much memory, to the point where the daily cron job was unable to run for long and some essential processes were OOM killed.

I thought 1GB was enough for a web process, as on the other unit each process was using less than 300MB, but then it seems mailman3 allows for very demanding requests... Sorry about that.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.