'apt-mark showauto' and 'apt-cache show' is slow

Everything in apt uses the cache and that has to be generated. Plus the depcache, I guess.

Do you say information about automatically installed packages can not be read from /var/lib/apt/extended_states (or whatever file set in configuration variable Dir::State::extended_states)? If I install something by apt, I see modification time of /var/lib/apt/extended_states about 1 sec later than /var/log/apt/term.log

It first builds or opens the cache, then the depcache, which reads extended_states. Just reading extended_states may be inaccurate.

Two problems: (1) depcache opening is slow (2) we don't really need the depcache here. Anyway, triaged, but low importance.

Well, I think it is not a problem, if we do not need depcache here :)

The problem is that the code that reads the state reads it into the depcache.

I'm not going to add a second independent reader to apt just so that apt-mark runs a bit faster.

I just found out information about packages installed from PPAs are not found in file configured in Dir::State::extended_states, so my script does not work for them.
Are they found in some other file then?

That's simply not true.

Maybe so, but the fact is that the file does not contain information about all packages installed to my system. (apt 2.0.4 (amd64) / Ubuntu 20.04)

$ grep -c '^Package:' /var/lib/apt/extended_states
698
$ dpkg -l | grep ^ii | wc -l
1893

My file contains only records for packages that are automatically installed for some reason. (All records have "Auto-Installed: 1")

apt-mark showmanual works anyway.

Maybe I should make another bug report about it?

There is no bug there, this is all working correctly.

So the file should only list packages that are installed automatically or what?

Yes

I wonder why didn't you choose to have only manually installed packages in the file? It would be even smaller file.

Mechanical train signals used to signal if the next section is clear vs. blocked by another train used to have the arm raised if it was clear and down if not. That was so that if the mechanic would fail in some way the arm would fall down and rest in the "blocked" state rather than in a "clear" state potentially causing huge problems.

I like to think that it is the same here. If the data gets lost, corrupted or whatever we fall back to a safe state: protected from autoremove as manual installed rather than causing havoc. In reality the original implementer might not have thought that far, but he isn't around anymore to ask him… and it isn't really important, is it?

Your precived "slowness" might be an out of dated cache. Have you recently run an apt command with root rights? If it is not run as root apt will build the cache in memory only. That cache can also easily be many megabytes big, which can take a while to shuffle into disk cache on a slow spinning disk.

Your script might be very fast, but it is also wrong (Auto-Installed is not the only field which can be set there even if for apt its the only one used. Other clients could use other fields and in that case apt would of course also keep the manual entries…) and doesn't even begin to support what apt-mark does. If you want to constructively work on speeding apt in these cases is to look at where the time is lost and optimize those codepaths. Showing off your script-foo is not helping and borderline trolling… after all, I can easily point at a lot of traveling salesperson (well, perhaps not in corona times), but that seems like a very hard problem for computers somehow.

Well, the script in the description is a bit too simple. In my application, I was looking for 'Auto-Installed:' field with value 0 to find manually installed packages, and due to the change in the way the file is build nowadays I do not find them by the code anymore. Now the field is pointless. I suppose now I could find the packages by set substraction, if I want to optimize my code.

"time apt-mark showauto >/dev/null"
is slow no matter when I run it. It is about the same if you run it for any single package. Do you call those a borderline cases? It might not be slow by latest and greatest supercomputers though, but I make code for slower hardware, too, and try to strive for reasonable responsiveness in my applications.

Oh, actually I was already using set substraction in my code, but I was also checking for 'Auto-Installed: 0'. Maybe that is necessary with some older version of apt? (There was some bug in my code and that is why it did not work correctly for some packages.) Sorry for blaming and confusing. However, I will change my code just in case you decide to remove the pointless Auto-Installed fields from the file.

It's slow because it loads the entire depcache, and checks all dependencies. This can be avoided by refactoring the code, but it seems unnecessary. I'd rather add useful features like telling you which packages become autoremovable garbage after marking something, rather than refactoring the code to not require a depcache.

Julian, BTW can you tell which version of apt was the latest where value of the Auto-Installed field can be something else than 1 in the file?

No such version exists as it would be a bug. An Auto-Installed field != 1 is still possible if the section includes another field the current apt version doesn't know about and hence can't reason about. apt itself does not currently generate such stanzas, but a future version might. Or other clients might.

Oh, currently missing Auto-Installed field seems to mean the package is manually installed even if the Package field exists. I suppose I can rely on that in the future.

It's also imaginable that we might change the value from e.g. 1 to yes. I don't know why it's an integer. In any case, I'd say you can't rely on that file at all. It might get changed in format, renamed or removed entirely without any notice.

Well, Boolean algebra may have had some influence on that. It is a common practice. 1 is shorter than yes.

Oh that is sad, because I am not satisfied with the speed of apt-mark. On the other hand it is good. Another format or data structure may be more efficient.

There is no performance issue with the file format, the parsing is not noticeable.

There is a massive design problem in the DepCache having to visit every dependency in the cache and do marking stuff. This worked fine for tiny sets, but scales superlinear to package count.

That's a very long time effort thing to fix for a future ABI break, we need to only check those dependencies we actually care about.

Out of those 0.5s, probably about .4s are spent checking dependencies. This will decrease by factor 10-50 presumably.

Good, besides simple text file format is easy to parse and check by command line tools, if needed.
To my surprise, I have more manually installed packages than automatically installed ones, so listing manually installed instead of automatically installed in the file would not give benefit.

'apt-mark showauto' and 'apt-mark showmanual' are simple filters that do not need any information about dependencies.

/var/lib/dpkg/status lists dependencies shown by 'apt-cache show'