DIRECTORY still resolves symlinks internally with :RESOLVE-SYMLINKS NIL

Bug #1114689 reported by Faré
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
SBCL
New
Undecided
Unassigned

Bug Description

With the attached patch, I could speed up (asdf:initialize-source-directory) by two orders of magnitude (!) when I (SETF ASDF:*RESOLVE-SYMLINKS* NIL).

I think that as a further cleanup,

1- the CLASSIFY-SYMLINKS flags might be renamed to RESOLVE-SYMLINKS, for consistency. Similarly, &KEY or &OPTIONAL might be used everywhere.

2- You might want to disable symlinks internally even when when RESOLVE-SYMLINKS is true, except at the end, to avoid needless computations.

3- You might want to have a cache (under user control) around some operations so that you don't call readlink and lstat and parse-unix-namestring over and over again on the same files, all the time. With proper caching, there's no reason why operations that return truename on a lot of files should be 2 orders of magnitude slower than without truename.

Revision history for this message
Faré (fahree) wrote :
Revision history for this message
Faré (fahree) wrote :

Oops, measuring the wrong thing. The speed up with RESOLVE-SYMLINKS NIL is much more modest than that. Oh well. Further optimizations might help '(see suggestions above).

Revision history for this message
Faré (fahree) wrote :

Note that even without wonderful speedups for ASDF, the patch is still a good idea, as it avoids unnecessary (indeed, unwanted) work.

Revision history for this message
Faré (fahree) wrote :

Once again this patch is still a good idea. Can some maintainer merge, test and commit?

Revision history for this message
pipping (pipping) wrote :

Update: This patch still applies cleanly to sbcl-1.3.9 and appears to fix an issue: Without it, as far as I can tell, there is currently no way to check for all subdirectories of a directory and exclude symlinks. With this patch, (directory "/path/to/dir/*/" :resolve-symlinks nil) gives you just that.

Revision history for this message
pipping (pipping) wrote :

Here's a test case. On Linux, I create the following set of files, links, and directories:

$ mkdir /home/pipping/sbcl-test
$ mkdir /home/pipping/sbcl-test/dir
$ mkdir /home/pipping/sbcl-test/.dir
$ mkdir /home/pipping/sbcl-test/dir/dir-in-dir
$ touch /home/pipping/sbcl-test/file
$ touch /home/pipping/sbcl-test/dir/file-in-dir
$ ln -s dir /home/pipping/sbcl-test/link-to-dir
$ ln -s file /home/pipping/sbcl-test/link-to-file
$ ln -s dir/dir-in-dir /home/pipping/sbcl-test/link-to-dir-in-dir
$

Here's an overview over the resulting situation:

$ ls -AlR /home/pipping/sbcl-test/
/home/pipping/sbcl-test/:
total 8
drwxrwxr-x 3 pipping pipping 4096 Sep 29 11:18 dir
drwxrwxr-x 2 pipping pipping 4096 Sep 29 11:05 .dir
-rw-rw-r-- 1 pipping pipping 0 Sep 29 11:06 file
lrwxrwxrwx 1 pipping pipping 3 Sep 29 11:06 link-to-dir -> dir
lrwxrwxrwx 1 pipping pipping 14 Sep 29 11:18 link-to-dir-in-dir -> dir/dir-in-dir
lrwxrwxrwx 1 pipping pipping 4 Sep 29 11:07 link-to-file -> file

/home/pipping/sbcl-test/dir:
total 4
drwxrwxr-x 2 pipping pipping 4096 Sep 29 11:18 dir-in-dir
-rw-rw-r-- 1 pipping pipping 0 Sep 29 11:06 file-in-dir

/home/pipping/sbcl-test/dir/dir-in-dir:
total 0

/home/pipping/sbcl-test/.dir:
total 0
$

Leaving files and links to files alone, we have
- two physical directories, dir and .dir
- two symlinks to directories, link-to-dir and link-to-dir-in-dir

With sbcl-1.3.9.108-e34dd00-linux-x64, I get:

* (directory "/home/pipping/sbcl-test/*/" :resolve-symlinks t)

(#P"/home/pipping/sbcl-test/.dir/" #P"/home/pipping/sbcl-test/dir/"
 #P"/home/pipping/sbcl-test/dir/dir-in-dir/")
*

This is not unexpected (dir could show up either once or twice here since link-to-dir points to dir).

* (directory "/home/pipping/sbcl-test/*/" :resolve-symlinks nil)

(#P"/home/pipping/sbcl-test/.dir/" #P"/home/pipping/sbcl-test/dir/"
 #P"/home/pipping/sbcl-test/link-to-dir-in-dir/"
 #P"/home/pipping/sbcl-test/link-to-dir/")
*

This is unexpected. The fact that #P"/home/pipping/sbcl-test/link-to-dir/" and #P"/home/pipping/sbcl-test/link-to-dir-in-dir/" are contained in this list means that SBCL must have internally dereferenced the link when I thought I had asked it not to. If I apply Faré's patch, I get this instead:

* (directory "/home/pipping/sbcl-test/*/" :resolve-symlinks t)

(#P"/home/pipping/sbcl-test/.dir/" #P"/home/pipping/sbcl-test/dir/"
 #P"/home/pipping/sbcl-test/dir/dir-in-dir/")
* (directory "/home/pipping/sbcl-test/*/" :resolve-symlinks nil)

(#P"/home/pipping/sbcl-test/.dir/" #P"/home/pipping/sbcl-test/dir/")
*

This makes more sense to me.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.