add autoremove to Landscape

Bug #1208393 reported by Alexander List
72
This bug affects 16 people
Affects Status Importance Assigned to Milestone
Landscape Client
Fix Committed
Undecided
Simon Poirier
Landscape Server
Fix Released
Wishlist
Simon Poirier
landscape-client (Ubuntu)
Fix Released
Wishlist
Andreas Hasenack
Trusty
Fix Released
Wishlist
Andreas Hasenack
Xenial
Fix Released
Wishlist
Andreas Hasenack
Zesty
Fix Released
Wishlist
Andreas Hasenack
Artful
Fix Released
Wishlist
Andreas Hasenack

Bug Description

[Impact]
It is good practice to remove unneeded packages/software from a machine. On the shell, this can be achieved by using "apt autoremove". There is no way to do it via landscape.

The lack of this option can lead to a full /boot partitions, for example, by accumulating kernel packages that are no longer used.

The fix in Landscape is a new checkbox option in the existing auto-upgrade profile feature. If checked, whenever an auto-upgrade profile is applied to a computer, meaning, upgrades will be done, the client will also do the equivalent of "apt autoremove", thus removing packages that are no longer used.

The caveat is that, at the time of this writing, the server portion of this feature is not yet available, so it can only be tested with a trunk deployment of the server component, or from staging.landscape.canonical.com as soon as it also gets this client update (yes, the client update is needed on the server as well for this feature to work).

[Test Case]

* get a test account on staging.landscape.canonical.com. That deployment has autoremove support. You may email the address in the #landscape topic in the internal canonical irc channel to request such an account.

* Login to https://staging.landscape.canonical.com and create an autoupgrade profile, configure it to apply to computers with the tag "autoupgraderemove", and make sure to tick the "autoremove packages" option. Also configure it to apply to all days of the week (or at least "today") and at every hour at XX minutes. You will change XX later.

* create an ubuntu VM or LXD (easier) running the release you are testing. On it, make sure there are upgrades available. Do not upgrade the packages. If there are no available upgrades (i.e., the image you picked to deploy this VM or LXD was up-to-date), inspect packages you have installed with apt-cache policy and try to downgrade some. You need at least one upgrade available.

* install devscripts, and then remove it:
  - sudo apt install devscripts
  - sudo apt purge devscripts
That should trigger a lot of autoremovable possibilities. Confirm by running (but not proceeding with) "apt autoremove"

* install landscape-client from proposed and register it against staging. Something like:
sudo landscape-config --silent -a <your-account-name> -t test-sru-1208393 --script-users ALL --include-manager-plugins ScriptExecution -u https://staging.landscape.canonical.com/message-system --ping-url http://staging.landscape.canonical.com/ping

* go to the staging.landscape.canonical.com pending computers page and accept this computer

* select the computer, go to the packages page and wait for this computer to report all packages and available upgrades. This could take about 15min or even more on artful because staging does not have a hash-id cache for artful.

* keep tailing -f /var/log/landscape/package-reporter.log on the client and look for multiple messages where it says it's reporting autoremovable packages. Here is one example:
2017-11-13 13:49:22,220 INFO [MainThread] Queuing message with changes in known packages: 421 installed, 62279 available, 44 available upgrades, 0 locked, 1 autoremovable, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable.

* That confirms the client is reporting the autoremovable packages.

* Once all packages have been reported to the server (no more "changes in known packages on the client log), on the server, add the tag "autoupgraderemove" to this registered computer.

* Edit the autoupgrade profile you created before and make sure it is configured to kick in in the next 5 minutes. Change its configuration if needed.

* Wait 5-10min. You should eventually see an activity that will upgrade packages on this computer, as well as autoremove others.

* Once the activity is finished, confirm that "apt autoremove" on the computer no longer has packages to remove.

[Regression Potential]
This feature relies on "apt autoremove" working correctly and not doing silly things, like removing your running kernel, or bash, or something else that's essential.

It's an opt-in feature, so it won't affect people who do not enable it in the auto-upgrade profiles.

[Other Info]
This is technically a new feature, but it has been one of the most requested ones.

As of the time of this writing, there is no public landscape server (apart from staging) which has support for this feature. But it's good to have the client out there for when the server is upgraded.

A new client talking to an old server won't cause problems. The server will simply discard the extra autoremovable package list.

This PPA has packages built from these branches:

https://launchpad.net/~ahasenack/+archive/ubuntu/lsclient-sru-1721383

They are using a ~ppaN suffix.

--- Original Description below ---

Canonical IS have moved software updates almost entirely to Landscape.

Apart from updating software packages, it is also good practice to remove unneeded software.

On the shell, this can be achieved using "apt-get autoremove"

It would be desirable to have this functionality available in Landscape.

Revision history for this message
Chris Glass (tribaal) wrote :

After some IRC discussion:

The autoremove run needs to be approved *per computer* (and AFAIK the current state lets admins approve on an activity basis only). Otherwise the "run script" feature would suffice.

Changed in landscape:
milestone: none → later
Benji York (benji)
Changed in landscape:
status: New → Triaged
Revision history for this message
Tim Kuhlman (timkuhlman) wrote :

An occasional apt-get clean would be nice also. It doesn't typically clean up as much as getting rid of a bunch of old kernels but does help a bit.

tags: added: is
Changed in landscape:
assignee: nobody → David Britton (davidpbritton)
Revision history for this message
James Troup (elmo) wrote :

FWIW, we just had a production server run out of inodes because
landscape kept installing new kernels but didn't ever run autoremove.

unable to create `/usr/src/linux-headers-3.13.0-91/arch/ia64/include/asm/kexec.h.dpkg-new' (while processing `./usr/src/linux-headers-3.13.0-91/arch/ia64/include/asm/kexec.h'): No space left on device

 Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda1 655360 651791 3569 100% /

Revision history for this message
Tom Haddon (mthaddon) wrote :

We're continuing to see this as an issue across VMs and physical servers of all kinds, and it needs manual clean up. This is affecting SREs and developers running services as well. Will be sure to raise this in the next IS/Landscape call to see if we can get the priority raised.

Revision history for this message
Ricardo Kirkner (ricardokirkner) wrote :

We just saw this on one of our services as well. No juju errors reported, all units seemed fine but the service was down. Upon closer inspection in the failing unit there seemed to be missing disk space (from the error on the terminal: bash: cannot create temp file for here-document: No space left on device).

du however reported lots of space left, and free also reported enough ram available. Checking for inodes reported 100% usage.

In order to fix this (the unit had more than a handful old kernels still lying around) we first mounted a tmpfs fs on /tmp to work around the issue of apt-get needing to write temporary files.

It looks liked the inode count got full while running some apt operation because apt-get autoremove wouldn't run even after solving the /tmp issue. A quick dpkg --configure -a and apt-get -f install allowed us to apt-get autoremove and get the inode count back down to a more acceptable 19%.

Revision history for this message
Robin Winslow (nottrobin) wrote :

We've also been hit by space running out in production and staging units in about 4 of our environments so far. It would be very helpful to have these old kernels automatically removed.

Revision history for this message
Tom Haddon (mthaddon) wrote :

One possibility for implementing this would be an option to an upgrade profile. There's currently an "Only security upgrades" option. If we also had a "Also run autoremove" option too that'd be great.

Revision history for this message
Matt Goodall (matt-goodall) wrote :

Just been hit with the same issue. Disk space was 61% but inode use was 100%. An `apt-get autoremove` removed 39 old linux-headers* and linux-image* packages after which the disk was usable again.

Revision history for this message
David Britton (dpb) wrote :

Idea from the CPT sprint is a checkbox in upgrade profiles "also auto remove packages", and a client change to run apt-get autoremove if that is specified.

Revision history for this message
David Britton (dpb) wrote :

@elmo -- what if we *always* autoremoved if a machine is covered by a upgrade-all-packages upgrade profile?

Revision history for this message
James Troup (elmo) wrote :

+1

(More fine grained control would be nice someday, but I'd happily accept blanket autoremove in the meantime.)

Paul Gear (paulgear)
tags: added: canonical-is
Changed in landscape:
assignee: David Britton (davidpbritton) → Данило Шеган (danilo)
David Britton (dpb)
Changed in landscape:
assignee: Данило Шеган (danilo) → nobody
Changed in landscape:
status: Triaged → Won't Fix
Revision history for this message
Junien Fridrick (axino) wrote :

Hi,

Why is this moved to "Won't fix" ? Just yesterday, we had a new package kernel installed that caused some servers to run out of disk space.

Changed in landscape:
status: Won't Fix → Triaged
Revision history for this message
Thomas Cuthbert (tcuthbert) wrote :

Hi,

We've received more alerts that servers have run out of disk space due to an excess of installed kernels.

Simon Poirier (simpoir)
Changed in landscape:
assignee: nobody → Simon Poirier (simpoir)
status: Triaged → In Progress
Changed in landscape:
status: In Progress → Fix Committed
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hi, I can't get this feature to work.

I deployed lds trunk, created the upgrade profile with the autoremove flag enabled, registered a client that had one upgrade available and dozens of autoremove packages, watched exchanges:

(...)
2017-11-10 11:45:45,277 INFO [MainThread] Queuing message with changes in known packages: 118 installed, 9998 available, 0 available upgrades, 0 locked, 13 autoremovable, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable.
2017-11-10 11:46:52,761 INFO [MainThread] Queuing message with changes in known packages: 100 installed, 8903 available, 0 available upgrades, 0 locked, 13 autoremovable, 0 not installed, 0 not available, 0 not available upgrades, 0 not locked, 0 not autoremovable.
(...)

But on the server the landscape-resource-1.computer_packages.autoremovable table row remained empty:
landscape-resource-1=# select available_upgrades,autoremovable from computer_packages;
 available_upgrades | autoremovable
--------------------+---------------
 {316069} | {}
(1 row)

When the upgrade time came, only the one package was upgraded, and the autoremovable ones remained installed.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

The problem was in the debdiff provided in the SRU bug, it missed a crucial hunk for this fix.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

landscape-client was fix committed in r1039 in LP, and https://github.com/CanonicalLtd/landscape-client/pull/11 upstream

information type: Proprietary → Public
Changed in landscape-client:
status: New → Fix Committed
assignee: nobody → Simon Poirier (simpoir)
Changed in landscape-client (Ubuntu):
status: New → In Progress
assignee: nobody → Andreas Hasenack (ahasenack)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 16.03-0ubuntu4

---------------
landscape-client (16.03-0ubuntu4) bionic; urgency=medium

  [ Simon Poirier]
  * Update landscape-client with upstream fixes (LP: #1721383):
    - Add proxy handling to package reporter. (LP: #1531150)
    - Fix regression in configuration hook under install-cd chroot (LP: #1699789)
    - Report autoremovable packages (LP: #1208393)
    - Do not re-register client by default (LP: #1618483)
    - Remove diff "noise" from d/p/ignore-backports-1668583.diff

 -- Andreas Hasenack <email address hidden> Fri, 10 Nov 2017 12:26:23 -0200

Changed in landscape-client (Ubuntu):
status: In Progress → Fix Released
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
Changed in landscape-client (Ubuntu Trusty):
status: New → In Progress
Changed in landscape-client (Ubuntu Xenial):
status: New → In Progress
Changed in landscape-client (Ubuntu Zesty):
status: New → In Progress
Changed in landscape-client (Ubuntu Artful):
status: New → In Progress
Changed in landscape-client (Ubuntu Trusty):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in landscape-client (Ubuntu Xenial):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in landscape-client (Ubuntu Zesty):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in landscape-client (Ubuntu Artful):
assignee: nobody → Andreas Hasenack (ahasenack)
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Alexander, or anyone else affected,

Accepted landscape-client into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/16.03-0ubuntu3.17.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in landscape-client (Ubuntu Artful):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-artful
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Alexander, or anyone else affected,

Accepted landscape-client into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/16.03-0ubuntu3.17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in landscape-client (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed-zesty
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Alexander, or anyone else affected,

Accepted landscape-client into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/16.03-0ubuntu2.16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in landscape-client (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Alexander, or anyone else affected,

Accepted landscape-client into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/landscape-client/14.12-0ubuntu6.14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in landscape-client (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed-trusty
Revision history for this message
David Coronel (davecore) wrote :

I tested upgrading the new landscape-client on artful, zesty, xenial and trusty from the current most recent landscape-client version currently available in each release. I didn't test this particular fix but I confirm the package upgrade works fine and the client runs fine afterwards.

Mathew Hodson (mhodson)
Changed in landscape-client (Ubuntu):
importance: Undecided → Wishlist
Changed in landscape-client (Ubuntu Trusty):
importance: Undecided → Wishlist
Changed in landscape-client (Ubuntu Xenial):
importance: Undecided → Wishlist
Changed in landscape-client (Ubuntu Zesty):
importance: Undecided → Wishlist
Changed in landscape-client (Ubuntu Artful):
importance: Undecided → Wishlist
Revision history for this message
Eric Desrochers (slashd) wrote :

Please test with the above [Test Case] in the description.

- Eric

Revision history for this message
Simon Poirier (simpoir) wrote :

I verified this with with a clean deploy of landscape-server-quickstart and packaging from ppa:landscape/lds-trunk and clients on trusty, xenial, zesty, artful. Packages marked auto were reported correctly in the various logs and were later removed by the profile.

Eric Desrochers (slashd)
tags: added: verification-done-artful verification-done-trusty verification-done-xenial verification-done-zesty
removed: verification-needed-artful verification-needed-trusty verification-needed-xenial verification-needed-zesty
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

The verification comment mentions using packages from ppa:landscape/lds-trunk instead of -proposed. Please make sure that the verification has been made on the packages that are in the proposed pocket of the main Ubuntu archives.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 16.03-0ubuntu3.17.10.1

---------------
landscape-client (16.03-0ubuntu3.17.10.1) artful; urgency=medium

  [ Simon Poirier ]
  * Add proxy handling to package reporter. (LP: #1531150)
  * Fix regression in configuration hook under install-cd chroot (LP: #1699789)
  * Report autoremovable packages (LP: #1208393)
  * No not re-register client by default (LP: #1618483)

 -- Andreas Hasenack <email address hidden> Fri, 10 Nov 2017 15:44:17 -0200

Changed in landscape-client (Ubuntu Artful):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for landscape-client has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 16.03-0ubuntu3.17.04.1

---------------
landscape-client (16.03-0ubuntu3.17.04.1) zesty; urgency=medium

  [ Simon Poirier ]
  * Add proxy handling to package reporter. (LP: #1531150)
  * Fix regression in configuration hook under install-cd chroot (LP: #1699789)
  * Report autoremovable packages (LP: #1208393)
  * No not re-register client by default (LP: #1618483)

 -- Andreas Hasenack <email address hidden> Fri, 10 Nov 2017 16:06:54 -0200

Changed in landscape-client (Ubuntu Zesty):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 16.03-0ubuntu2.16.04.2

---------------
landscape-client (16.03-0ubuntu2.16.04.2) xenial; urgency=medium

  [ Simon Poirier ]
  * Add proxy handling to package reporter. (LP: #1531150)
  * Fix regression in configuration hook under install-cd chroot (LP: #1699789)
  * Report autoremovable packages (LP: #1208393)
  * No not re-register client by default (LP: #1618483)

 -- Andreas Hasenack <email address hidden> Fri, 10 Nov 2017 16:09:30 -0200

Changed in landscape-client (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 14.12-0ubuntu6.14.04.1

---------------
landscape-client (14.12-0ubuntu6.14.04.1) trusty; urgency=medium

  [ Simon Poirier ]
  * Add proxy handling to package reporter. (LP: #1531150)
  * Fix regression in configuration hook under install-cd chroot (LP: #1699789)
  * Report autoremovable packages (LP: #1208393)
  * No not re-register client by default (LP: #1618483)

 -- Andreas Hasenack <email address hidden> Fri, 10 Nov 2017 16:21:54 -0200

Changed in landscape-client (Ubuntu Trusty):
status: Fix Committed → Fix Released
Simon Poirier (simpoir)
Changed in landscape:
milestone: later → 18.03
Changed in landscape:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.