Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Misc static improvements (mainly ocaml and a regression linked to b0b5ef7286dca098f40f5075175105c3c0dfbe05) #133008

Merged
merged 35 commits into from
Aug 20, 2021

Conversation

symphorien
Copy link
Member

Motivation for this change

Bulding static ocaml packages

Most of the PR should be uncontroversial, but there is a change to stdenv linked to a regression in b0b5ef7

The gist is .overrideAttrs (o: { dontAddStaticConfigureFlags = true }) does not work because o.configureFlags already contains the faulty flags and they will not be remove any more. You can see this on current nixpkgs with:

nix show-derivation $(nix-instantiate . -A pkgsStatic.ocaml-ng.ocamlPackages_4_07.findlib) | jq '.[].env'

which has both dontAddStaticConfigureFlags set to 1 and --disable-shared in configureFlags.
(this affects all of ocaml packages)
Proposed solution is to move the logic from nix (which is already complex with all the overrideScope' at play) to bash in stdenv builder.

cc @alyssais for dontAddStaticConfigureFlags and @veprbl for ocaml part.

Tested with

nix build -f. {pkgsStatic.,}ocaml-ng.ocamlPackages_4_{06,07,08,09,10,11,12}.{dune_2,ocaml,ocamlgraph}

This is a mass rebuild, so please refrain from bikeshedding on spacing as it takes half a day to rebuild.

@symphorien symphorien force-pushed the ocaml-static branch 2 times, most recently from e77f227 to fcf470a Compare August 7, 2021 11:45
@ofborg ofborg bot added 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild 10.rebuild-linux-stdenv This PR causes stdenv to rebuild labels Aug 7, 2021
@ofborg ofborg bot requested review from edolstra and lovek323 August 7, 2021 13:37
@veprbl veprbl requested a review from vbgl August 7, 2021 15:22
@symphorien symphorien added the 6.topic: cross-compilation Building packages on a different platform than they will be used on label Aug 7, 2021
Copy link
Contributor

@Mindavi Mindavi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't lnow much about the stdenv stuff, but other than 1 comment LGTM. Have not tried building.

@@ -7,7 +7,7 @@
, gnutlsSupport ? false, gnutls ? null
, wolfsslSupport ? false, wolfssl ? null
, scpSupport ? zlibSupport && !stdenv.isSunOS && !stdenv.isCygwin, libssh2 ? null
, gssSupport ? with stdenv.hostPlatform; !(
, gssSupport ? with stdenv.hostPlatform; (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this change. Why invert it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before the change:
gss support is enabled when: on windows, or building a static library or cross compiling to darwin. Given the issues linked in comment, this looks backward to me.

@SuperSandro2000
Copy link
Member

Can we set some strictDeps to prevent regressions in the future?

@symphorien
Copy link
Member Author

done

@Ericson2314
Copy link
Member

I like where are you are going with this :). Two things, and then it's good

  1. You should fix dontAddStaticConfigureFlags by instead making the stdenv adapter also override the definition overrideAttrs. The problem is overrideAttrs closes over the old mkDerivation. I don't like seeing new stuff in setup.sh because that file has a terrible history of only growing more complex, never shrinking.

  2. ocamlFixPackage ocamlStaticAdapter barely do anything now, can you just kill them off entirely? I want to get rid of the stdenv adapter completely and you've already done the hard underlying work of getting us closer :).

@symphorien
Copy link
Member Author

symphorien commented Aug 12, 2021 via email

@symphorien
Copy link
Member Author

I attempted something. I suspect that topkg-based packages require more work for real cross to work. Compile tested by

nix build -f. {pkgsStatic.,}ocaml-ng.ocamlPackages_4_{06,07,08,09,10,11,12}.{merlin,ocamlgraph}

@symphorien
Copy link
Member Author

@GrahamcOfBorg eval

the ci failure is a flaky test in aws-c-common: #133857 (comment)

@ofborg ofborg bot requested a review from marsam August 16, 2021 20:20
@Ericson2314
Copy link
Member

@symphorien Nice attempt. That does work, but I think it will be even easier if we make mkDerivation an argument to stdenv with a default argument. Then we can do

stdenv.override (old: {
  mkDerivation = ... old.mkDerivation ...;
})

and it will "just work", no mapMakeDerivation... needed.

Mind giving that a shot?

@Ericson2314
Copy link
Member

oh it might have to be mkDerivationFromStdenv so you can tie that knot properly late too. I would also make make-derivation.nix just return the function not { mkDerivation = ...; } while you are at it, so that file ends up looking like:

{ lib, config }:

stdenv:

/* mkDerivation lambda */

The override would be

stdenv.override (old: {
  mkDerivationFromStdenv = ... old.mkDerivationFromStdenv ...;
})

and default arg

{
...
, mkDerivationFromStdenv ? import ./make-derivation.nix { inherit lib config; }
}: 
...

@Ericson2314
Copy link
Member

I ended up just doing the adapter fixing part of this. I'll put up a PR tomorrow which can just go to master. Then this can just worry about the individual packages on top.

@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Aug 19, 2021
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Aug 19, 2021
@Ericson2314
Copy link
Member

Looks great!

@Ericson2314
Copy link
Member

Is this tested enough to merge?

@github-actions github-actions bot added the 6.topic: stdenv Standard environment label Aug 19, 2021
@Ericson2314
Copy link
Member

I added a commit getting rid of more things in the overlay. I also built that stuff.

@Ericson2314
Copy link
Member

Sorry I made a typo, should be fixed now.

Ericson2314 and others added 2 commits August 19, 2021 21:55
findlib's setup hook is now a preConfigure setup hook, so make sure to
run it
@symphorien
Copy link
Member Author

I checked that ocamlbuild and pkgsStatic.ocamlbuild build, execute and are from the expected platform. i won't have time to check much more.

@ofborg ofborg bot requested a review from peti August 19, 2021 23:36
@Ericson2314
Copy link
Member

I built those two in the end after the stuff I changed. Will merge this once ofborg finally signs off!

@ofborg ofborg bot requested a review from Ericson2314 August 20, 2021 00:35
@Ericson2314 Ericson2314 merged commit a54bafe into NixOS:staging Aug 20, 2021
@Ericson2314
Copy link
Member

Thanks again!

@vcunat
Copy link
Member

vcunat commented Sep 7, 2021

FYI, I'm not sure where in master..staging-next it broke, but pkgsStatic.curl won't build now. (undefined references to some brotli symbols; .curlMinimal does build)

@vcunat
Copy link
Member

vcunat commented Sep 7, 2021

Builds again after commit 4215591.

@rb2k
Copy link
Contributor

rb2k commented Sep 19, 2021

This commit: b1b2305#diff-db67b092b9f267a4586cc73d414de28f03cc5a5d774d23284c4bc745420ab8a7 breaks libbfd on x86 darwin on master from what I can tell.

Without the added 'libbfd: add strictDeps = true' line, it compiles fine.
With the line, I end up with:

/nix/store/j4bf4z7zrd461r4bjiwsrnzc5am0jv9a-bash-4.4-p23/bin/bash ./libtool  --tag=CC   --mode=link clang -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow  -g -O2  -release `cat libtool-soversion` -liberty   -o libbfd.la -rpath /nix/store/sbmig002wpjdzafrjhnil8bx97pdq28h-libbfd-2.35.1/lib archive.lo archures.lo bfd.lo bfdio.lo bfdwin.lo cache.lo coff-bfd.lo compress.lo corefile.lo elf-properties.lo format.lo hash.lo init.lo libbfd.lo linker.lo merge.lo opncls.lo reloc.lo section.lo simple.lo stab-syms.lo stabs.lo syms.lo targets.lo binary.lo ihex.lo srec.lo tekhex.lo verilog.lo `cat ofiles` -liberty -lintl -ldl -lz 
libtool: link: clang -dynamiclib -Wl,-undefined -Wl,dynamic_lookup -o .libs/libbfd-2.35.1.dylib  .libs/archive.o .libs/archures.o .libs/bfd.o .libs/bfdio.o .libs/bfdwin.o .libs/cache.o .libs/coff-bfd.o .libs/compress.o .libs/corefile.o .libs/elf-properties.o .libs/format.o .libs/hash.o .libs/init.o .libs/libbfd.o .libs/linker.o .libs/merge.o .libs/opncls.o .libs/reloc.o .libs/section.o .libs/simple.o .libs/stab-syms.o .libs/stabs.o .libs/syms.o .libs/targets.o .libs/binary.o .libs/ihex.o .libs/srec.o .libs/tekhex.o .libs/verilog.o .libs/mach-o-x86-64.o .libs/mach-o-i386.o .libs/mach-o.o .libs/dwarf2.o .libs/pef.o .libs/xsym.o .libs/plugin.o .libs/elf32-aarch64.o .libs/elf64-aarch64.o .libs/elfxx-aarch64.o .libs/aix5ppc-core.o .libs/aout64.o .libs/coff-alpha.o .libs/coff-x86_64.o .libs/coff64-rs6000.o .libs/elf32-ia64.o .libs/elf32-mips.o .libs/elf32-score.o .libs/elf32-score7.o .libs/elf64-alpha.o .libs/elf64-gen.o .libs/elf64-hppa.o .libs/elf64-ia64.o .libs/elf64-ia64-vms.o .libs/elfxx-ia64.o .libs/elfn32-mips.o .libs/elf64-mips.o .libs/elfxx-mips.o .libs/elf64-mmix.o .libs/elf64-nfp.o .libs/elf64-ppc.o .libs/elf32-riscv.o .libs/elf64-riscv.o .libs/elfxx-riscv.o .libs/elf64-s390.o .libs/elf64-sparc.o .libs/elf64-tilegx.o .libs/elf64-x86-64.o .libs/elfxx-x86.o .libs/elf64-bpf.o .libs/elf64.o .libs/mach-o-aarch64.o .libs/mmo.o .libs/pe-x86_64.o .libs/pei-ia64.o .libs/pei-x86_64.o .libs/pepigen.o .libs/pex64igen.o .libs/vms-alpha.o .libs/aout-cris.o .libs/aout-ns32k.o .libs/aout32.o .libs/cf-i386lynx.o .libs/coff-go32.o .libs/coff-i386.o .libs/coff-mips.o .libs/coff-rs6000.o .libs/coff-sh.o .libs/coff-stgo32.o .libs/coff-tic30.o .libs/coff-tic4x.o .libs/coff-tic54x.o .libs/coff-z80.o .libs/coff-z8k.o .libs/coffgen.o .libs/cofflink.o .libs/dwarf1.o .libs/ecoff.o .libs/ecofflink.o .libs/elf-attrs.o .libs/elf-eh-frame.o .libs/elf-ifunc.o .libs/elf-m10200.o .libs/elf-m10300.o .libs/elf-nacl.o .libs/elf-strtab.o .libs/elf-vxworks.o .libs/elf.o .libs/elf32-am33lin.o .libs/elf32-arc.o .libs/elf32-arm.o .libs/elf32-avr.o .libs/elf32-bfin.o .libs/elf32-cr16.o .libs/elf32-cris.o .libs/elf32-crx.o .libs/elf32-csky.o .libs/elf32-d10v.o .libs/elf32-d30v.o .libs/elf32-dlx.o .libs/elf32-epiphany.o .libs/elf32-fr30.o .libs/elf32-frv.o .libs/elf32-ft32.o .libs/elf32-gen.o .libs/elf32-h8300.o .libs/elf32-hppa.o .libs/elf32-i386.o .libs/elf32-ip2k.o .libs/elf32-iq2000.o .libs/elf32-lm32.o .libs/elf32-m32c.o .libs/elf32-m32r.o .libs/elf32-m68hc11.o .libs/elf32-m68hc12.o .libs/elf32-m68hc1x.o .libs/elf32-m68k.o .libs/elf32-s12z.o .libs/elf32-mcore.o .libs/elf32-mep.o .libs/elf32-metag.o .libs/elf32-microblaze.o .libs/elf32-moxie.o .libs/elf32-msp430.o .libs/elf32-mt.o .libs/elf32-nds32.o .libs/elf32-nios2.o .libs/elf32-or1k.o .libs/elf32-pj.o .libs/elf32-ppc.o .libs/elf32-pru.o .libs/elf32-rl78.o .libs/elf32-rx.o .libs/elf32-s390.o .libs/elf32-sh.o .libs/elf32-sparc.o .libs/elf32-spu.o .libs/elf32-tic6x.o .libs/elf32-tilegx.o .libs/elf32-tilepro.o .libs/elf32-v850.o .libs/elf32-vax.o .libs/elf32-visium.o .libs/elf32-wasm32.o .libs/elf32-xc16x.o .libs/elf32-xgate.o .libs/elf32-xstormy16.o .libs/elf32-xtensa.o .libs/elf32-z80.o .libs/elf32.o .libs/elflink.o .libs/elfxx-sparc.o .libs/elfxx-tilegx.o .libs/i386aout.o .libs/i386bsd.o .libs/i386lynx.o .libs/i386msdos.o .libs/mach-o-arm.o .libs/ns32knetbsd.o .libs/pc532-mach.o .libs/pdp11.o .libs/pe-arm-wince.o .libs/pe-arm.o .libs/pe-i386.o .libs/pe-mcore.o .libs/pe-ppc.o .libs/pe-sh.o .libs/pei-arm-wince.o .libs/pei-arm.o .libs/pei-i386.o .libs/pei-mcore.o .libs/pei-ppc.o .libs/pei-sh.o .libs/peigen.o .libs/ppcboot.o .libs/reloc16.o .libs/som.o .libs/vax1knetbsd.o .libs/vaxnetbsd.o .libs/vms-lib.o .libs/vms-misc.o .libs/wasm-module.o .libs/xcofflink.o .libs/xtensa-isa.o .libs/xtensa-modules.o .libs/cpu-i386.o .libs/cpu-powerpc.o .libs/cpu-rs6000.o .libs/cpu-aarch64.o .libs/cpu-alpha.o .libs/cpu-arc.o .libs/cpu-arm.o .libs/cpu-avr.o .libs/cpu-bfin.o .libs/cpu-bpf.o .libs/cpu-cr16.o .libs/cpu-cris.o .libs/cpu-crx.o .libs/cpu-csky.o .libs/cpu-d10v.o .libs/cpu-d30v.o .libs/cpu-dlx.o .libs/cpu-epiphany.o .libs/cpu-fr30.o .libs/cpu-frv.o .libs/cpu-ft32.o .libs/cpu-h8300.o .libs/cpu-hppa.o .libs/cpu-iamcu.o .libs/cpu-l1om.o .libs/cpu-k1om.o .libs/cpu-ia64.o .libs/cpu-ip2k.o .libs/cpu-iq2000.o .libs/cpu-lm32.o .libs/cpu-m10200.o .libs/cpu-m10300.o .libs/cpu-m32c.o .libs/cpu-m32r.o .libs/cpu-m68hc11.o .libs/cpu-m68hc12.o .libs/cpu-m9s12x.o .libs/cpu-s12z.o .libs/cpu-m9s12xg.o .libs/cpu-m68k.o .libs/cpu-mcore.o .libs/cpu-mep.o .libs/cpu-metag.o .libs/cpu-microblaze.o .libs/cpu-mips.o .libs/cpu-mmix.o .libs/cpu-moxie.o .libs/cpu-msp430.o .libs/cpu-mt.o .libs/cpu-nds32.o .libs/cpu-nfp.o .libs/cpu-nios2.o .libs/cpu-ns32k.o .libs/cpu-or1k.o .libs/cpu-pdp11.o .libs/cpu-pj.o .libs/cpu-pru.o .libs/cpu-riscv.o .libs/cpu-rl78.o .libs/cpu-rx.o .libs/cpu-s390.o .libs/cpu-score.o .libs/cpu-sh.o .libs/cpu-sparc.o .libs/cpu-spu.o .libs/cpu-tic30.o .libs/cpu-tic4x.o .libs/cpu-tic54x.o .libs/cpu-tic6x.o .libs/cpu-tilegx.o .libs/cpu-tilepro.o .libs/cpu-v850.o .libs/cpu-v850_rh850.o .libs/cpu-vax.o .libs/cpu-visium.o .libs/cpu-wasm32.o .libs/cpu-xc16x.o .libs/cpu-xgate.o .libs/cpu-xstormy16.o .libs/cpu-xtensa.o .libs/cpu-z80.o .libs/cpu-z8k.o .libs/archive64.o   -liberty -lintl -ldl -lz  -g -O2   -install_name  /nix/store/sbmig002wpjdzafrjhnil8bx97pdq28h-libbfd-2.35.1/lib/libbfd-2.35.1.dylib  -Wl,-single_module
ld: library not found for -lintl
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [Makefile:1437: libbfd.la] Error 1
make[2]: Leaving directory '/private/tmp/nix-build-libbfd-2.35.1.drv-0/binutils-2.35.1/bfd'
make[1]: *** [Makefile:1807: all-recursive] Error 1
make[1]: Leaving directory '/private/tmp/nix-build-libbfd-2.35.1.drv-0/binutils-2.35.1/bfd'
make: *** [Makefile:1330: all] Error 2
builder for '/nix/store/fvqxn7n351mwsjh1isfarfc58jqz4g16-libbfd-2.35.1.drv' failed with exit code 2
error: build of '/nix/store/fvqxn7n351mwsjh1isfarfc58jqz4g16-libbfd-2.35.1.drv' failed

@vbgl
Copy link
Contributor

vbgl commented Nov 4, 2021

Did this break OCaml 4.09 on darwin? https://hydra.nixos.org/build/152737288/nixlog/3/tail

Did this break OCaml on darwin aarch64? https://hydra.nixos.org/build/157047473/nixlog/2/tail

@symphorien
Copy link
Member Author

symphorien commented Nov 4, 2021

This kind of error is caused by a missing

hardeningDisable = [ "strictoverflow" ];

(see https://nixos.org/manual/nixpkgs/stable/#strictoverflow )
This PR does not change oveflow flags so probably it is not the cause.
I can make a PR to disable that, but I cannot test so I'd rather let someone else with access to a mac do it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants