dlltool uses non-unique temp filenames

With binutils-2.38 in a cross to x86_64-w64-mingw32 I persistently see random breakage in dlltool during the build of mingw-w64's "crt".

Example 1:
...
x86_64-w64-mingw32-dlltool --as-flags=--64 -m i386:x86-64 -k --as=x86_64-w64-mingw32-as --output-lib lib64/libd3dcompiler_33.a --input-def /tmp/mingw-w64-v9.0.0/mingw-w64-crt/lib64/d3dcompiler_33.def
x86_64-w64-mingw32-dlltool --as-flags=--64 -m i386:x86-64 -k --as=x86_64-w64-mingw32-as --output-lib lib64/libd3dcompiler_34.a --input-def /tmp/mingw-w64-v9.0.0/mingw-w64-crt/lib64/d3dcompiler_34.def
x86_64-w64-mingw32-dlltool --as-flags=--64 -m i386:x86-64 -k --as=x86_64-w64-mingw32-as --output-lib lib64/libd3dcompiler_35.a --input-def /tmp/mingw-w64-v9.0.0/mingw-w64-crt/lib64/d3dcompiler_35.def
x86_64-w64-mingw32-dlltool --as-flags=--64 -m i386:x86-64 -k --as=x86_64-w64-mingw32-as --output-lib lib64/libd3dcompiler_36.a --input-def /tmp/mingw-w64-v9.0.0/mingw-w64-crt/lib64/d3dcompiler_36.def
Assembler messages:
Error: can't open D3DCompiler_dll_t.s for reading: No such file or directory
x86_64-w64-mingw32-dlltool --as-flags=--64 -m i386:x86-64 -k --as=x86_64-w64-mingw32-as --output-lib lib64/libd3dcompiler_37.a --input-def /tmp/mingw-w64-v9.0.0/mingw-w64-crt/lib64/d3dcompiler_37.def
x86_64-w64-mingw32-dlltool: x86_64-w64-mingw32-as exited with status 1
x86_64-w64-mingw32-dlltool: failed to open temporary tail file: D3DCompiler_dll_t.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00000.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00001.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00002.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00003.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00004.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00005.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00006.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00007.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00008.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00009.o: No such file or directory
make[1]: *** [Makefile:83854: lib64/libd3dcompiler_36.a] Error 1
make[1]: *** Waiting for unfinished jobs....
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00000.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00001.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00002.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00003.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00004.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00005.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_dll_s00006.o: No such file or directory
x86_64-w64-mingw32-dlltool: cannot delete D3DCompiler_d...

I think the problem is that multiple instances of dlltool may use the same temporary file names depending on the files being built.

With fdeee5d59dca41e3c70c399a939105e39a4b4282, the temporary file prefix will be taken from the .def file, and the .def files may not have unique names. For example, in mingw-w64-crt/lib64, d3dcompiler_{33,34,35,36}.def all have the same LIBRARY name inside: "D3DCompiler.dll" [1].

I've also seen this bug when building Wine (on both Linux and macOS), where multiple dlltools may be launched at the same time to build delay-load and cross libraries for the same DLL (using the same .spec as input).

I'm not sure of the best fix for this: maybe add a command-line option to use a deterministic tmp_prefix, with the caveat that parallel builds may fail?

(As an aside, I'm not sure why one would want deterministic tmp file names. Maybe for reproducible builds?)

[1]: https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-crt/lib64/d3dcompiler_33.def

(In reply to Brendan Shanks from comment #1)
> With fdeee5d59dca41e3c70c399a939105e39a4b4282, the temporary file prefix
> will be taken from the .def file, and the .def files may not have unique
> names. For example, in mingw-w64-crt/lib64, d3dcompiler_{33,34,35,36}.def
> all have the same LIBRARY name inside: "D3DCompiler.dll" [1].

Sorry about this - I really didn't forsee that this would be an issue.

> I've also seen this bug when building Wine (on both Linux and macOS), where
> multiple dlltools may be launched at the same time to build delay-load and
> cross libraries for the same DLL (using the same .spec as input).

In wine, what cases of multiple parallel import libraries targeting the same DLL name?

> I'm not sure of the best fix for this: maybe add a command-line option to
> use a deterministic tmp_prefix, with the caveat that parallel builds may
> fail?

That doesn't sound very ideal to me. Ideally it would do the right thing, deterministically, out of the box, without conflict.

If the issue is that the deterministic temp file names can clash when running multiple dlltool invocations in parallel, should we perhaps create a separate temp directory (where the name can be nondeterministic), where we make sure we get a unique directory that nobody else is using, then write all temp files within that one?

Or should we make the deterministic temp prefix based on something else, e.g. the ouput (path+)filename? That should be unique even with multiple jobs running in parallel, if they're running in the same directory. The temp prefix base might end up slightly longer, so it's maybe a little less elegant, but should be much safer against clashes.

> (As an aside, I'm not sure why one would want deterministic tmp file names.
> Maybe for reproducible builds?)

Msys2 had issues with the nondeterministic nature of dlltool. In most cases, the temp prefix doesn't matter - but for umbrella libraries like libucrt.a, which is merged from a number of individual import libraries like this, https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-crt/lib-common/ucrt.mri, it matters: The temp prefix for all the individual libraries making up libucrt.a must be unique, otherwise ld.bfd mixes up the object files for those libraries, producing intermixed import tables for those DLLs. See https://github.com/msys2/MINGW-packages/issues/9363#issuecomment-899100856 for the root cause.

We worked around this with https://github.com/mingw-w64/mingw-w64/commit/0ad299826ca14987fd53664c1047f4dfe848c3f8, which adds the --temp-prefix option (based on the output import library name, which should be unique) if the option is supported.

We also noticed that Debian had been using the --temp-prefix option in this way to make their builds reproducible: https://salsa.debian.org/mingw-w64-team/mingw-w64/-/blob/master/debian/patches/dlltool-temp-prefix.patch

Based on this, I thought it'd be good if we could make dlltool fix both of these issues out of the box, so that users of it don't need to take extra steps to both achieve reproducibility and avoid the intermixed import tables.

(Also, if building a newer version of mingw-w64 that contains the commit above, which ...

Created attachment 14013
Potential fix

Here's a pretty straightforward patch that uses the output filename as basis for the temp prefix, which I think should avoid this issue (if there's no other aspect I'm overlooking). Does this fix the issue for you?

(In reply to Martin Storsjö from comment #3)
> Created attachment 14013 [details]
> Potential fix
>
> Here's a pretty straightforward patch that uses the output filename as basis
> for the temp prefix, which I think should avoid this issue (if there's no
> other aspect I'm overlooking). Does this fix the issue for you?

Thanks Martin, that worked for me for a Wine build. Mikael, can you try building mingw-w64's crt?

(In reply to Brendan Shanks from comment #4)
> (In reply to Martin Storsjö from comment #3)
> > Created attachment 14013 [details]
> > Potential fix
> >
> > Here's a pretty straightforward patch that uses the output filename as basis
> > for the temp prefix, which I think should avoid this issue (if there's no
> > other aspect I'm overlooking). Does this fix the issue for you?
>
> Thanks Martin, that worked for me for a Wine build. Mikael, can you try
> building mingw-w64's crt?

I managed to reproduce the issue building mingw-w64-crt, and the patch does fix that case too. (I could also verify that building the latest git master version of mingw-w64-crt isn't affected by the bug, as those versions explicitly specify unique --temp-prefix strings.)

I'll go ahead and send the patch for review then.

(In reply to Brendan Shanks from comment #4)
> (In reply to Martin Storsjö from comment #3)
> > Created attachment 14013 [details]
> > Potential fix
> >
> > Here's a pretty straightforward patch that uses the output filename as basis
> > for the temp prefix, which I think should avoid this issue (if there's no
> > other aspect I'm overlooking). Does this fix the issue for you?
>
> Thanks Martin, that worked for me for a Wine build. Mikael, can you try
> building mingw-w64's crt?

After applying the proposed fix on top of binutils-2.38 I can no longer reproduce the problem.

The master branch has been updated by Nick Clifton <email address hidden>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=d65c0ddddd85645cab6f11fd711d21638a74489f

commit d65c0ddddd85645cab6f11fd711d21638a74489f
Author: Martin Storsj? <email address hidden>
Date: Wed Mar 16 15:01:30 2022 +0000

    dlltool: Use the output name as basis for deterministic temp prefixes

            PR 28885
            * dlltool.c (main): use imp_name rather than dll_name when
            generating a temporary file name.

Fixed on master for 2.39, needs backport to binutils-2_38-branch.

The binutils-2_38-branch branch has been updated by Nick Clifton <email address hidden>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=99852365513266afdd793289813e8e565186c9e6

commit 99852365513266afdd793289813e8e565186c9e6
Author: Nick Clifton <email address hidden>
Date: Wed Mar 23 11:39:49 2022 +0000

    dlltool: Use the output name as basis for deterministic temp prefixes

            PR 28885
            * dlltool.c (main): use imp_name rather than dll_name when
            generating a temporary file name.

And now fixed on binutils-2_38-branch, closing.

Building wine 7.5 (with wine-staging patches, but they shouldn't be relevant to this) with "make -j64" fails with

tools/winebuild/winebuild -b x86_64-w64-mingw32 -w --implib -o dlls/rpcrt4/librpcrt4.delay.a --export \
  /home/bero/abf/wine/BUILD/wine-7.5/dlls/rpcrt4/rpcrt4.spec
/usr/bin/x86_64-w64-mingw32-dlltool: bfd_open failed reopen stub file: rpcrt4_dll_s00176.o: No such file or directory
winebuild: /usr/bin/x86_64-w64-mingw32-dlltool failed with status 1
make: *** [Makefile:297034: dlls/rpcrt4/librpcrt4.delay.a] Error 1

Probably the library is assembled before all object files belonging to it are built.

Building with make (without SMP) works.

Is there a change if you build with lesser threads? Eg. make -j8 or something assuming you have a threadripper with 32 cores/64 threads?

And possibly also interesting - distro, gcc and mingw-w64 version.

These objects file are not created from the makefile. It sounds more like it's running into some kind of resource limit.

(In reply to Alexandre Julliard from comment #2)
> These objects file are not created from the makefile. It sounds more like
> it's running into some kind of resource limit.

tmpfs maybe? Just wondering since i got that exact error when building wine on OBS.. Also got a couple of other very strange errors that failed building on OBS with files that was not found, but could not find a error message from the actual compile..

I've run into a similar issue

looking at the compile traces lets me believe that the issue arises when two
instances of mingw dlltool run at the same time, and thrashing each other temporary files

indeed, this ugly hack lets the compilation succeeds (it was failing almost always; sometimes on a different DLL)
-----------
diff --git a/tools/winebuild/import.c b/tools/winebuild/import.c
index c876d51f8e6..8049530a7e5 100644
--- a/tools/winebuild/import.c
+++ b/tools/winebuild/import.c
@@ -1595,7 +1595,11 @@ static void build_windows_import_lib( const char *lib_na>
     strarray_add( &args, lib_name );
     strarray_add( &args, "-d" );
     strarray_add( &args, def_file );
-
+ strarray_add( &args, "-t" );
+ {
+ char tmp[128]; sprintf(tmp, "%u\n", getpid());
+ strarray_add( &args, tmp );
+ }
     switch (target.cpu)
     {
         case CPU_i386:
----------
Bernhard, can you test this on your side?

I don't see a simple way to fix it...

hmmm thinking about it, doesn't look quite right (or sufficient)

in the dlltool source code, if no -t prefix is given, will generate its own prefix based on pid...
need more investigation

I'm testing on two environments:
E1) with dlltool 2.38-1
E2) with dlltool 2.37-3

I see the error in E1 only <g>

on all env, I run
> strace winebuild... <c/l from makefile> -v -v >& log
> grep exec log

results:
====================
E2)
pid 12272] execve("/usr/bin/x86_64-w64-mingw32-dlltool", ["/usr/bin/x86_64-w64-mingw32-dllt"..., "-k", "-y", "dlls/iphlpapi/libiphlpapi.delay."..., "-d", "libiphlpapi.delay-625082dd.def", "-m", "i386:x86-64", "--as-flags=--64"], 0x7ffc19487b48 /* 56 vars */) = 0
[pid 12273] execve("/usr/bin/x86_64-w64-mingw32-as", ["/usr/bin/x86_64-w64-mingw32-as", "--64", "-o", "daesh.o", "daesh.s"], 0x7fffa1e12cf8 /* 56 vars */ <unfinished ...>
[pid 12273] <... execve resumed>) = 0
[pid 12274] execve("/usr/bin/x86_64-w64-mingw32-as", ["/usr/bin/x86_64-w64-mingw32-as", "--64", "-o", "daest.o", "daest.s"], 0x7fffa1e12cf8 /* 56 vars */ <unfinished ...>
[pid 12274] <... execve resumed>) = 0

====================
E1-without the patch below)
execve("tools/winebuild/winebuild", ["tools/winebuild/winebuild", "-b", "x86_64-w64-mingw32", "-w", "--implib", "-o", "dlls/iphlpapi/libiphlpapi.delay."..., "--export", "/home/eric/wine/wine/dlls/iphlpa"..., "-v", "-v"], 0x7ffc4d598b28 /* 54 vars */) = 0
[pid 9078] execve("/usr/bin/x86_64-w64-mingw32-dlltool", ["/usr/bin/x86_64-w64-mingw32-dllt"..., "-k", "-y", "dlls/iphlpapi/libiphlpapi.delay."..., "-d", "libiphlpapi.delay-6250734a.def", "-m", "i386:x86-64", "--as-flags=--64"], 0x7ffd66663948 /* 54 vars */) = 0
[pid 9079] execve("/usr/bin/x86_64-w64-mingw32-as", ["/usr/bin/x86_64-w64-mingw32-as", "--64", "-o", "iphlpapi_dll_h.o", "iphlpapi_dll_h.s"], 0x7ffd4c406e08 /* 54 vars */ <unfinished ...>
[pid 9079] <... execve resumed>) = 0
[pid 9080] execve("/usr/bin/x86_64-w64-mingw32-as", ["/usr/bin/x86_64-w64-mingw32-as", "--64", "-o", "iphlpapi_dll_t.o", "iphlpapi_dll_t.s"], 0x7ffd4c406e08 /* 54 vars */ <unfinished ...>
[pid 9080] <... execve resumed>) = 0

====================
E1-with the patch above)
execve("tools/winebuild/winebuild", ["tools/winebuild/winebuild", "-b", "x86_64-w64-mingw32", "-w", "--implib", "-o", "dlls/iphlpapi/libiphlpapi.delay."..., "--export", "/home/eric/wine/wine/dlls/iphlpa"..., "-v", "-v"], 0x7ffd5925c228 /* 54 vars */) = 0
[pid 9601] execve("/usr/bin/x86_64-w64-mingw32-dlltool", ["/usr/bin/x86_64-w64-mingw32-dllt"..., "-k", "-y", "dlls/iphlpapi/libiphlpapi.delay."..., "-d", "libiphlpapi.delay-6250774f.def", "-t", "9600\n", "-m", "i386:x86-64", "--as-flags=--64"], 0x7fff84ee6468 /* 54 vars */) = 0
[pid 9602] execve("/usr/bin/x86_64-w64-mingw32-as", ["/usr/bin/x86_64-w64-mingw32-as", "--64", "-o", "9600\nh.o", "9600\nh.s"], 0x7ffdc7cf0c98 /* 54 vars */ <unfinished ...>
[pid 9602] <... execve resumed>) = 0
[pid 9603] execve("/usr/bin/x86_64-w64-mingw32-as", ["/usr/bin/x86_64-w64-mingw32-as", "--64", "-o", "9600\nt.o", "9600\nt.s"], 0x7ffdc7cf0c98 /* 54 vars */ <unfinished ...>
[pid 9603] <... execve resumed>) = 0

to summarize, binutils 2.38 no longer generates a temp file name dependant on the pid... using the -t options forces back the regular behavior...

Bernhard, Sveinar: could you also look at the dlltool --version ...

https://sourceware.org/bugzilla/show_bug.cgi?id=28885

so it's a known issue in 2.38 and fixed in 2.39
backport to 2.38 to be checked

(In reply to Eric Pouech from comment #7)
> https://sourceware.org/bugzilla/show_bug.cgi?id=28885
>
> so it's a known issue in 2.38 and fixed in 2.39
> backport to 2.38 to be checked

So, this seems to be the patch then? https://sourceware.org/git/?p=binutils-gdb.git;a=patch;h=99852365513266afdd793289813e8e565186c9e6

binutils-38 for jammy was updated 23/3-22, but could not find anything in the debian patches nor the changelog for this.

> binutils-38 for jammy was updated 23/3-22, but could not find anything in
> the debian patches nor the changelog for this.
that's what I meant with "backport to 2.38 to be checked"... not sure <put here your favorite distro> will post an update for this change

(In reply to Eric Pouech from comment #9)
> > binutils-38 for jammy was updated 23/3-22, but could not find anything in
> > the debian patches nor the changelog for this.
> that's what I meant with "backport to 2.38 to be checked"... not sure <put
> here your favorite distro> will post an update for this change

and to be precise, one should check the binutils version on which mingw's dlltool is built upon (not the binutils version for the ELF compilation, which may be on a different version)

x86_64-w64-mingw32-dlltool --version
i686-w64-mingw32-dlltool --version

I can confirm that recompiling mingw-binutils witht his patch https://sourceware.org/git/?p=binutils-gdb.git;a=patch;h=99852365513266afdd793289813e8e565186c9e6
fix the issue

*** Bug 53011 has been marked as a duplicate of this bug. ***

Resolving upstream.

*** Bug 53024 has been marked as a duplicate of this bug. ***

I just ran into this trying to build wine-devel-7.9 packages for jammy. The Ubuntu bug is https://bugs.launchpad.net/ubuntu/+source/binutils-mingw-w64/+bug/1971901.

*** Bug 53077 has been marked as a duplicate of this bug. ***

*** Bug 53675 has been marked as a duplicate of this bug. ***