Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report outputs that are larger than Hydra's size limit #398

Open
mschwaig opened this issue Apr 28, 2024 · 1 comment
Open

Report outputs that are larger than Hydra's size limit #398

mschwaig opened this issue Apr 28, 2024 · 1 comment

Comments

@mschwaig
Copy link

Issue

Hydra has a size limit for outputs, which means that outputs larger than 3 GB are not cached and the Hydra job that builds them will fail.

Proposed Solution

I think nixpkgs-review should report packages that build successfully, but have such derivations in their runtime closure as failing.

  • Ideally with exceptions of some form around derivations with the preferLocalBuild or allowSubstitutes attribute.
  • The reason for the failure should be listed in the report files.

Example Occurrence

As an example, this could have caught my change here NixOS/nixpkgs#287846, which made a part of ROCm larger than Hydra's size limit.

Related issues

This issue is different from #36, because it is about the limits Hydra imposes on individual derivations, and not about keeping an eye on the overall size of the closure.

Current state /Alternatives

Right now a rough check of this limit can be done manually on the results of nixpkgs-review like this:

mschwaig@hydralisk ~/.c/n/pr-298388-9> cd results/
mschwaig@hydralisk ~/.c/n/p/results> du -h --apparent-size --max-depth 0 */ | grep G
3.7G	rocmPackages.composable_kernel/
1.6G	rocmPackages.miopen/
3.1G	rocmPackages.rocblas/
2.1G	rocmPackages.rocfft/
1.5G	rocmPackages.rocsolver/
1.2G	rocmPackages.rocsparse/

original comment

It's a bit tricky because the --apparent-size flag is really important on file systems with compression like ZFS, and it takes some effort to find out how Hydra actually measures size and what the configured limit is.

@Mic92
Copy link
Owner

Mic92 commented May 9, 2024

Sounds good. We can skip those warnings however for unfree software.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants