bintools: Add response file support to ld-wrapper

[Gabriella439]

The motivation behind this is to alleviate the problem described in #41340. I'm not sure if this completely fixes the problem, but it eliminates one more area where we can exceed command line length limits.

This is essentially the same change as in #112449, except for ld-wrapper.sh instead of cc-wrapper.sh.

However, that change alone was not enough; on macOS the ld provided by darwin.cctools fails if you use process substitution to generate the response file, so I put up a PR to fix that:

tpoechtrager/cctools-port#131

… and I included a patch referencing that fix so that the new ld-wrapper still works on macOS.

I verified that the ld provided by .#bintools now correctly handles response files without a segmentation fault

Things done

• Built on platform(s)
  • aarch64-darwin
• Tested basic functionality of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

[Gabriella439]

For convenience, here is the relevant excerpt of the build failure from the eval:

error: builder for '/nix/store/a1qy9fhlggpwsz71kcc2pqy5b481cgx3-aws-c-common-0.8.9.drv' failed with exit code 2;
       last 10 log lines:
       > 423/423 Test #309: device_rand_u16_distribution .........................................   Passed    6.65 sec
       >
       > 99% tests passed, 1 tests failed out of 423
       >
       > Total Test time (real) =   8.33 sec
       >
       > The following tests FAILED:
       >   118 - bus_async_test_churn (SEGFAULT)
       > Errors while running CTest
       > make: *** [Makefile:71: test] Error 8
       For full logs, run 'nix log /nix/store/a1qy9fhlggpwsz71kcc2pqy5b481cgx3-aws-c-common-0.8.9.drv'.

… which appears to be a flaky test failure for an unrelated test that just happened to be downstream of this change: https://github.com/awslabs/aws-c-common/blob/8c51377ddff0aea9e21ecd9f58e9a4d283539751/tests/bus_test.c#L354-L436

[trofi]

LGTM

Very nice fix! Survived a few thousand packages on x86_64-linux from my system as well.

[Gabriella439]

Since this is deep in the dependency tree, do I need to select another base branch to merge into? I don't know exactly what is the policy for PRs like these

(I'm still doing some additional testing before merging this, too, on our internal codebase)

[trofi]

Ah, yes. It should be rebased against staging. Otherwise it should be OK.

[trofi]

Eval failed with:

The following tests FAILED:
        118 - bus_async_test_churn (SEGFAULT)
Errors while running CTest
make: *** [Makefile:71: test] Error 8
error: builder for '/nix/store/mgcshrniwk02rqv9mzr6vzqq4d9g6zsb-aws-c-common-0.8.9.drv' failed with exit code 2;

Not sure if by related.

[trofi]

@ofborg eval

[Gabriella439]

Yeah, I think that test is flaky and just happens to be downstream of this change. It failed earlier on this PR, then passed, then failed again

[Gabriella439]

So I found the root cause of the failures, which I fixed in 1d9ab50. It turns out the failures were indicative of a real problem which is gone now.

I've also stress tested this on our internal Haskell codebase which has a large number of dependencies (both Haskell and non-Haskell dependencies) and is also itself a Haskell project with ~5000 modules (so, a lot to link in a single command, which is what motivated this fix in the first place). I verified that with this change and https://gitlab.haskell.org/ghc/ghc/-/merge_requests/9897 we no longer hit the ARG_MAX linker limit due to consistent use of response files across the stack.

There was one failure that ofborg reported (on x86_64-darwin) that seems to just be a timeout, but the other platforms built fine, so I'm inclined to merge once the upstream PR against cctools-port (tpoechtrager/cctools-port#132) is merged and then I'll adjust this PR to update the patch URL before merging.