"apt-cache search" is slow because of listing /var/lib/apt/lists 45227 times (triggered by 'Acquire::Languages "none";')

Bug #1004947 reported by Daniel Hahler
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
apt (Debian)
Fix Released
Unknown
apt (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

I've recognized that "apt-cache search" has become quite slow on my system, and when stracing it it shows that it lists the contents of /var/lib/apt/lists a lot (45227 times!):

    openat(AT_FDCWD, "/var/lib/apt/lists/", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 3
    getdents64(3, /* 156 entries */, 32768) = 13920
    getdents64(3, /* 0 entries */, 32768) = 0
    close(3) = 0

% grep -c 'openat(AT_FDCWD, "/var/lib/apt/lists/"' apt-cache-search-strace.txt
45227
(this count is reproducible with a second run)

% time apt-cache search tomahawk
apt-cache search tomahawk 3.84s user 4.54s system 99% cpu 8.427 total

I am attaching the full output of strace.

TEST CASE:
% strace apt-cache search tomahawk 2>&1 | grep -c 'openat(AT_FDCWD, "/var/lib/apt/lists/"'
45227

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: apt 0.8.16~exp12ubuntu10
ProcVersionSignature: Ubuntu 3.2.0-24.39-generic-pae 3.2.16
Uname: Linux 3.2.0-24-generic-pae i686
ApportVersion: 2.0.1-0ubuntu8
Architecture: i386
Date: Sat May 26 19:39:11 2012
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release i386 (20110427.1)
SourcePackage: apt
UpgradeStatus: Upgraded to precise on 2011-12-16 (162 days ago)

CVE References

Revision history for this message
Daniel Hahler (blueyed) wrote :
Daniel Hahler (blueyed)
description: updated
Changed in apt (Debian):
status: Unknown → New
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt (Ubuntu):
status: New → Confirmed
Revision history for this message
Marcos Magalhães (marcos-daekdroom) wrote :

Mistakenly set it as affecting me, but it doesn't. The test case returns '1' for me. So I'm setting it back to 'New'.

Changed in apt (Ubuntu):
status: Confirmed → New
Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :

Hi Daniel,
  I can't get this to trigger on my 12.04 system, could you include a copy of your /etc/apt/apt.conf.d and /etc/apt/preferences.d?
Another suggestion you could try would be to install the ddebs and then run apt-cache search under gdb and ctrl-c it somewhere in the middle, hope you hit the slow patch and then try and get a backtrace to see how it got there.

Dave

Revision history for this message
Daniel Hahler (blueyed) wrote :

When running it in gdb, I get the following backtrace:
Starting program: /usr/bin/apt-cache search apt
^C
Program received signal SIGINT, Interrupt.
0xb7fdd424 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7fdd424 in __kernel_vsyscall ()
#1 0xb7c9c70f in __getdents64 (fd=16304, buf=<optimized out>, nbytes=<optimized out>) at ../sysdeps/unix/sysv/linux/getdents.c:156
#2 0xb7c9c977 in __readdir64 (dirp=0x806d1d8) at ../sysdeps/unix/readdir.c:66
#3 0xb7f58d5a in APT::Configuration::getLanguages(bool const&, bool const&, char const**) () from /usr/lib/i386-linux-gnu/libapt-pkg.so.4.12
#4 0xb7edf650 in pkgCache::VerIterator::TranslatedDescription() const () from /usr/lib/i386-linux-gnu/libapt-pkg.so.4.12
#5 0x08051bd1 in Search (CmdL=...) at apt-cache.cc:1273
#6 0xb7ecaf6e in CommandLine::DispatchArg(CommandLine::Dispatch*, bool) () from /usr/lib/i386-linux-gnu/libapt-pkg.so.4.12
#7 0x0804e833 in main (argc=3, argv=0xbfffdf84) at apt-cache.cc:1802

So it looks like the iterator used to get translated descriptions causes this.

This gets triggered by the following configuration:
% cat /etc/apt/apt.conf.d/20notranslation
Acquire::Languages "none";

summary: "apt-cache search" is slow because of listing /var/lib/apt/lists 45227
- times
+ times (triggered by 'Acquire::Languages "none";')
Changed in apt (Debian):
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (22.3 KiB)

This bug was fixed in the package apt - 0.9.6ubuntu1

---------------
apt (0.9.6ubuntu1) quantal-proposed; urgency=low

  [ Michael Vogt ]
  * merged from Debian, remaining changes:
    - use ubuntu keyring and ubuntu archive keyring in apt-key
    - run update-apt-xapian-index in apt.cron
    - support apt-key net-update and verify keys against master-keyring
    - run apt-key net-update in cron.daily
    - different example sources.list
    - APT::pkgPackageManager::MaxLoopCount set to 5000
    - apport pkgfailure handling
    - ubuntu changelog download handling
    - patch for apt cross-building, see http://bugs.debian.org/666772

  [ Steve Langasek ]
  * Drop upgrade handling for obsolete conffile /etc/apt/apt.conf.d/01ubuntu,
    removed in previous LTS.
  * prepare-release: declare the packages needed as source build deps.

apt (0.9.6) unstable; urgency=low

  [ David Kalnischkies ]
  * apt-pkg/cdrom.cc:
    - fix regression from 0.9.3 which dumped the main configuration
      _config instead of the cdrom settings (Cnf) as identified and
      tested by Milan Kupcevic, thanks! (Closes: #674100)
  * cmdline/apt-get.cc:
    - do not show 'list of broken packages' header if no package
      is broken as it happens e.g. for external resolver errors
    - print URIs for all changelogs in case of --print-uris,
      thanks to Daniel Hartwig for the patch! (Closes: #674897)
    - show 'bzr branch' as 'bzr get' is deprecated (LP: #1011032)
    - check build-dep candidate if install is forbidden
  * debian/apt-utils.links:
    - the internal resolver 'apt' is now directly installed in
      /usr/lib/apt/solvers, so don't instruct dh to create a broken link
  * doc/apt-verbatim.ent:
    - APT doesn't belong to the product 'Linux', so use 'APT' instead
      as after all APT is a big suite of applications
  * doc/examples/sources.list:
    - use the codename instead of 'stable' in the examples sources.list
      as we do in the manpage and as the debian-installer does
  * doc/apt-get.8.xml:
    - use apt-utils as package example instead of libc6
  * apt-pkg/contrib/cmdline.cc:
    - apply patch from Daniel Hartwig to fix a segfault in case
      the LongOpt is empty (Closes: #676331)
    - fix segfault with empty LongOpt in --no-* branch
  * ftparchive/apt-ftparchive.cc:
    - default to putting the Contents-* files below $(SECTION) as apt-file
      expects them there - thanks Martin-Éric Racine! (Closes: #675827)
  * apt-pkg/deb/deblistparser.cc:
    - set pkgCacheGen::Essential to "all" again (Closes: #675449)
  * apt-pkg/algorithms.cc:
    - force install only for one essential package out of a group
  * apt-pkg/aptconfiguration.cc:
    - if APT::Languages=none save "none" in allCodes so that the detected
      configuration is cached as intended (Closes: #674690, LP: #1004947)
  * apt-pkg/cacheiterators.h:
    - add an IsMultiArchImplicit() method for Dep- and PrvIterator

  [ Justin B Rye ]
  * doc/apt-cdrom.8.xml:
    - replace CDROM with the proper CD-ROM in text
    - correct disc vs. disk issues
  * doc/apt-extracttemplates.1.xml:
    - debconf is not DebConf
  * doc/apt-get.8.xml:
    - move dselect-upgrade below dist-upgrade
    - rev...

Changed in apt (Ubuntu):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.