Too many duplicates in hsc command line arguments, causing build failures due to GCC argument limits

[robbert-vdh]

I couldn't find an open bug report for this. Feel free to close this as a duplicate if it already exists!

Describe the bug
When cabal invokes hsc, it builds a list of arguments that among other things, include linker arguments for linking against all of the package's (transitive) dependencies' C dependencies:

cabal/Cabal/src/Distribution/Simple/PreProcess.hs

Lines 609 to 622 in e7657ad

	++ [ "--lflag=" ++ opt
	| pkg <- pkgs
	, opt <-
	["-L" ++ opt | opt <- Installed.libraryDirs pkg]
	++ [ "-Wl,-R," ++ opt | isELF, opt <- Installed.libraryDirs pkg
	]
	++ [ "-l" ++ opt
	| opt <-
	if withFullyStaticExe lbi
	then Installed.extraLibrariesStatic pkg
	else Installed.extraLibraries pkg
	]
	++ Installed.ldOptions pkg
	]

Because a lot of packages tend to depend on the same set of libraries, this ends up adding many duplicates of the same argument. Normally this doesn't cause any real problems because gcc and the linker will ignore these duplicates, but GCC also has an argument length limit. If you have enough dependencies then you will eventually trip this limit (see NixOS/nixpkgs#41340, though this hsc issue is a separate issue).

In a codebase I'm working on right now, hsc adds hundreds of the same arguments, causing it to trip this limit. This is an xcerpt from the the gcc command line, split on spaces, sorted by the number of occurrences:

...
      5 -L/nix/store/63w485iy0i6xkbdh2cr0sq98dbbj863v-tzdata-2024a-dev/lib
      5 -L/nix/store/971g0fka30djkcww2ddpx7q66dr86c1q-zlib-1.3.1/lib
      5 -L/nix/store/dq03m14rqjvbz6pw42g555df4khh4bvr-zlib-1.3.1-dev/lib
      5 -Wl,-R,/nix/store/63w485iy0i6xkbdh2cr0sq98dbbj863v-tzdata-2024a-dev/lib
      5 -Wl,-R,/nix/store/971g0fka30djkcww2ddpx7q66dr86c1q-zlib-1.3.1/lib
      5 -Wl,-R,/nix/store/dq03m14rqjvbz6pw42g555df4khh4bvr-zlib-1.3.1-dev/lib
      7 -L/nix/store/sk5if9vmsykj16phb87wv7a8pv08wi62-ghc-9.6.5/lib
      7 -Wl,-R,/nix/store/sk5if9vmsykj16phb87wv7a8pv08wi62-ghc-9.6.5/lib
    269 -L/nix/store/qi2525fzsnmgayxba5gxc2xi3x80g99x-ncurses-6.4.20221231/lib
    269 -Wl,-R,/nix/store/qi2525fzsnmgayxba5gxc2xi3x80g99x-ncurses-6.4.20221231/lib
    270 -L/nix/store/3bffbi88dmkpr5ib58wsngqd6bfdjzgj-elfutils-0.191/lib
    270 -L/nix/store/nj9g42fdsm8l2z43kfcahch3px2q209a-libffi-3.4.6/lib
    270 -L/nix/store/vgzmgmppvpb6gqlc56jhcn007cjn7xg9-gmp-with-cxx-6.3.0/lib
    270 -Wl,-R,/nix/store/3bffbi88dmkpr5ib58wsngqd6bfdjzgj-elfutils-0.191/lib
    270 -Wl,-R,/nix/store/nj9g42fdsm8l2z43kfcahch3px2q209a-libffi-3.4.6/lib
    270 -Wl,-R,/nix/store/vgzmgmppvpb6gqlc56jhcn007cjn7xg9-gmp-with-cxx-6.3.0/lib

In this case, the cabal build command terminates with the following error:

Build profile: -w ghc-9.6.5 -O1
In order, the following will be built (use -v for more details):
 - libname-0 (lib) (first run)
Preprocessing library for libname-0..
linking /repo/dist-newstyle/build/x86_64-linux/ghc-9.6.5/libname-0/build/libname/Constant/Buffers_hsc_make.o failed (exit code 1)
rsp file was: "/repo/dist-newstyle/build/x86_64-linux/ghc-9.6.5/libname-0/build/libname/Constant/hsc2hscall2431973-2.rsp"
command was: /nix/store/mpm3i0sbqc9svfch6a17179fs64dz2kv-gcc-wrapper-13.3.0/bin/cc <260 KiB worth of command line arguments>
error: gcc: fatal error: cannot execute ‘/nix/store/zc0nsv23pakbafngjy32kvhfzb16as43-gcc-13.3.0/libexec/gcc/x86_64-unknown-linux-gnu/13.3.0/collect2’: execv: Argument list too long
compilation terminated.

Error: .cabal-wrapped_: Failed to build libname-0.

I think simply deduplicating these arguments should fix the issue.

To Reproduce
I sadly cannot provide a reproducer because the project in question is not public, but this will happen on any project with a sufficiently large amount of dependencies. The problem is exacerbated by using Nix, since the paths to the library directories end up being much longer that way.

Expected behavior
c2hs/hsc should compile the module as expected.

System information

• Ubuntu 22.04
• nixpkgs pin 1e3deb3d8a86a870d925760db1a5adecc64d329d
• GHC 9.6.5 provided through Nix
• Cabal 3.10.3.0 provided through Nix

Additional information
I can provide a patch for this if needed. There may also be other command line invocations with similar problems.

[tbidne]

Also see NixOS/nixpkgs#41340.