-
-
Notifications
You must be signed in to change notification settings - Fork 13.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[wip] haskell: try to fix #41340 #56608
Conversation
For clarification, this cannot fundamentally fix the problem inherent in GCC's linker I describe in #41340 (comment) (which is that the amount of flags you can pass to GCC must not exceed 128 KB; which means that if you have 1000 Haskell dependencies ala If that is correct, reducing flags is still very useful for project that are somewhere below this upper bound; could you put a rough number estimate on how many flags we save with this PR? |
Yeah there is definitely still a limit, but I don't think we ever hit it before 18.03. My understanding is this was a regression that only came up by 18.03. I think we can fix it by just eliminating all duplication, and I believe this haskell stuff cut it down by ~30%. But, I'm also very skeptical of your claim that it has to be under 128KB. If true, that would mean that the Linux + GCC actually has a lower maximum argument count than macOS. In practice, I think I've only seen Linux hit it on Linux when we were still getting the 6x duplication. Almost every time this breaks macOS before it breaks Linux. |
@matthewbauer I've made a repro in Python on my Ubuntu 16.04: import os
import sys
# num_chars = 128000 # works
# num_chars = (2**16) # works
# num_chars = (2**17) # fails with E2BIG (Argument list too long)
# num_chars = (2**17 - 3) # works
num_chars = (2**17 - 2) # fails with E2BIG (Argument list too long)
sys.stdout.write("trying with " + str(num_chars) + " chars\n")
os.execve('/bin/true', [], {
'x': 'a' * num_chars
}) This shows
and
confirming the 128KB limit for me. Can you try this too? |
Yeah, I'm seeing the same limit. Interesting that we aren't hitting it more often. On macOS you don't hit this at all because clang isn't setting an environment variable like gcc. When we did hit this on macOS, it was always at linking, but I suspect that binutils doesn't pass those through an env var avoiding this limit. Do you have an example package where you hit this on Linux? |
Not any more. I have a project with 360 recursive Haskell dependencies total, which had hit the limit originally, but some of the previous flag deduplication improvements (see also #49206 (comment)) brought some integer factor improvements that brought it back under the limit. I expect to hit it again as the project grows. |
Motivation for this change
Attempt to fix #41340.
Things done
sandbox
innix.conf
on non-NixOS)nix-shell -p nox --run "nox-review wip"
./result/bin/
)nix path-info -S
before and after)