Extreme RAM and SWAP usage

apport information

apport information

Note that the kernel modules listed in the bug may be out of date - I don't have those modules listed, but i hadn't rebooted since they were installed, due to the antivirus in place on this server where I reported from (it's a mail server, so an antivirus engine with amavisd was necessary; note that the usage values for this bug report - for landscape client consuming lots of RAM - had the kernel modules 'off' and disabled with its standard software that correspond with the antivirus engine.)

I discussed this briefly in #ubuntu-hardened and #ubuntu-server and confirmed that the code hasn't been 'reviewed' lately.

I'm setting the importance here to "high" because I just got nipped with this again, and my primary mail server went down because of insanely high swap and RAM usage from Landscape Client. I'm going to test this with a new server to rule out headaches, but if landscape client is going to consume ungodly amounts of RAM, then it should be shelved until the memory leaks are solved.

Status changed to 'Confirmed' because the bug affects multiple users.

Also affecting me -- landscape-client using 25% of RAM on a 1GB server.

This is actually so severe that I started to regularly restart my small instances *every week*. The instance below is with an uptime of nine (9) days on a heavily used file server, and as you can see the first none landscape related process is the ssh-session I'm in! Sorted on memory %.

  PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
 1410 root 20 0 302M 229M 2252 S 0.0 25.2 35:39.09 /usr/bin/python /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/landscape-client.p
 1426 root 20 0 205M 117M 3768 S 0.0 12.9 9:08.72 /usr/bin/python /usr/bin/landscape-manager --ignore-sigint --quiet
 1470 landscape 20 0 552M 67320 5136 S 0.0 7.2 2:19.10 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1474 landscape 20 0 552M 67320 5136 S 0.0 7.2 2:18.09 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1423 landscape 20 0 552M 67320 5136 S 0.0 7.2 15:38.76 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1472 landscape 20 0 355M 28884 3780 S 0.0 3.1 0:00.01 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
 1424 landscape 20 0 355M 28884 3780 S 0.0 3.1 16:14.51 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
65097 root 20 0 119M 7772 6516 S 0.0 0.8 0:00.10 sshd: flindeberg [priv]

Same instance after reboot (sorted on memory %):

  PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
  976 mongodb 20 0 959M 62340 31504 S 0.0 6.7 0:00.00 /usr/bin/mongod --unixSocketPrefix=/run/mongodb --config /etc/mongodb.conf
  -- some other db and file server processes --
 1269 landscape 20 0 307M 54100 12616 S 0.0 5.8 0:00.49 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1231 landscape 20 0 307M 54100 12616 S 0.0 5.8 0:01.49 /usr/bin/python /usr/bin/landscape-broker --ignore-sigint --quiet
 1271 landscape 20 0 253M 38316 14680 S 0.0 4.1 0:00.01 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
 1232 landscape 20 0 253M 38316 14680 S 0.0 4.1 0:01.16 /usr/bin/python /usr/bin/landscape-monitor --ignore-sigint --quiet
 1233 root 20 0 104M 34472 11840 S 0.0 3.7 0:00.86 /usr/bin/python /usr/bin/landscape-manager --ignore-sigint --quiet
 1224 root 20 0 94364 24656 3236 S 1.3 2.6 0:00.32 /usr/bin/python /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/landscape-client.p

If it is of any use, all my instances are virtual, and I'd like to keep them really small.

I am still able to reproduce this issue with the following versions:

landscape-client: 16.03-0ubuntu2.16.04.3
landscape-server: 17.03.4-0ubuntu1
OS version: Ubuntu 16.04.4 LTS

Attached you can find a graph of roughly 24 hours. The container runs just the landscape client, otherwise it's the default ubuntu:lts image from LXD with only the snapd package removed.

Memory usage started with 180MB and raised up to around 480MB after 20 hours. See the attached image for details.

+1 for a fix here ... been forced to basically abandon landscape due to this bug .... glad to provide any info necessary for troubleshooting

When will the fix be available and for which distros?

https://github.com/CanonicalLtd/landscape-client/commit/7f98306e3159e956798b70d2b5fc8c0773373fdd

Is there an ETA for this fix? I really want to start using landscape again..

Can we please get an ETA for this, it is getting a little silly now, how you can possibly have your own management client have a bug fixed of this magnitude and it not be released is beyond me.

Since people continue to ask; We don't give ETAs on bug reports. Please mark this bug as affecting you if it is.

See: https://help.ubuntu.com/community/ReportingBugs, Etiquite Section, #9

Contribution in the form of testing help is always welcome (you can find the fix in the attached link in comment 12).

Thanks!

This bug was fixed in the package landscape-client - 18.01-0ubuntu7

---------------
landscape-client (18.01-0ubuntu7) disco; urgency=medium

  * debian/patches/mem-usage-1685885.patch: Reduce memory usage by
    unused twisted logs. (LP: #1685885)

 -- Simon Poirier <email address hidden> Fri, 08 Feb 2019 12:02:17 -0500

Hello Thomas, or anyone else affected,

Accepted landscape-client into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/18.01-0ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Thomas, or anyone else affected,

Accepted landscape-client into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/18.01-0ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Thomas, or anyone else affected,

Accepted landscape-client into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/16.03-0ubuntu2.16.04.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I tested this on xenial, bionic and cosmic by launching 2 identical lxd containers of each, one of which using the landscape-client/common from -proposed packages. Clients were registered in against the same account and left idle for a few hours before taking a reading:

  ps o rss,args `pgrep landscape-cli`

Here are the results which, as per test case, show the most noticeable improvement on xenial.

xenial-updates
  RSS COMMAND
254704 /usr/bin/python /usr/bin/landscape-client --daemon --pid-file /var/run/

xenial-proposed
  RSS COMMAND
46040 /usr/bin/python /usr/bin/landscape-client --daemon --pid-file /var/run/l

bionic-updates
  RSS COMMAND
33744 /usr/bin/python3 /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/

bionic-proposed
  RSS COMMAND
30960 /usr/bin/python3 /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/

cosmic-updates
  RSS COMMAND
33660 /usr/bin/python3 /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/la

cosmic-proposed
  RSS COMMAND
30912 /usr/bin/python3 /usr/bin/landscape-client --daemon --pid-file /var/run/landscape/la

The verification of the Stable Release Update for landscape-client has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package landscape-client - 18.01-0ubuntu4.2

---------------
landscape-client (18.01-0ubuntu4.2) cosmic; urgency=medium

  * debian/patches/mem-usage-1685885.patch: Reduce memory usage by
    unused twisted logs. (LP: #1685885)

 -- Simon Poirier <email address hidden> Fri, 08 Feb 2019 12:02:17 -0500

This bug was fixed in the package landscape-client - 18.01-0ubuntu3.3

---------------
landscape-client (18.01-0ubuntu3.3) bionic; urgency=medium

  * debian/patches/mem-usage-1685885.patch: Reduce memory usage by
    unused twisted logs. (LP: #1685885)

 -- Simon Poirier <email address hidden> Fri, 08 Feb 2019 12:02:17 -0500

This bug was fixed in the package landscape-client - 16.03-0ubuntu2.16.04.6

---------------
landscape-client (16.03-0ubuntu2.16.04.6) xenial; urgency=medium

  * debian/patches/mem-usage-1685885.patch: Reduce memory usage by
    unused twisted logs. (LP: #1685885)

 -- Simon Poirier <email address hidden> Fri, 08 Feb 2019 12:02:17 -0500