Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross Compiling Haskell to Windows #36200

Open
1 of 7 tasks
angerman opened this issue Mar 2, 2018 · 19 comments
Open
1 of 7 tasks

Cross Compiling Haskell to Windows #36200

angerman opened this issue Mar 2, 2018 · 19 comments
Labels
6.topic: cross-compilation Building packages on a different platform than they will be used on 6.topic: haskell 6.topic: windows Running, or buiding, packages on Windows

Comments

@angerman
Copy link
Contributor

angerman commented Mar 2, 2018

As some may already know, I'm working on cross compiling Haskell to Windows using nix.

I'll try to keep track of the issues I run into, and then add the relevant PRs as I manage to
clean them up.

Minor

  • mingw64 is too old for GHC. GHC requires mingw-w64 >= 5
  • cctools asserts targetPlatform.isDarwin, which fails.
  • ghc binary expressions assert stdenv.targetPlatform == stdenv.hostPlatform, which fails during validation.
  • The buildMK in the ghc expressions is wrong and needs to also include the relevant $flavour.mk
  • The generic builder uses hsc2hs from the nativeGhc, but should use $targetPrefix-hsc2hs.
  • haskellSetupDepends should come from the buildPackages package set as they are compiled and run by the native compiler.

Major

  • packageOverrides and buildPackages don't play nice together. Injecting a custom ghc via config doesn't work anymore, as buildPackages is fixed prior to the overrides being applied, and buildPackages doesn't see any added packages from the packageOverrides.

I might be missing some philosophical foundation with respect to nix here. Please bear with me and help me better understand.

/cc @Ericson2314

@Ericson2314 Ericson2314 added 6.topic: cross-compilation Building packages on a different platform than they will be used on 6.topic: haskell 6.topic: windows Running, or buiding, packages on Windows labels Mar 2, 2018
@Ericson2314
Copy link
Member

Ericson2314 commented Mar 2, 2018

mingw64 is too old for GHC. GHC requires mingw-w64 >= 5

👍 Definitely you can just bump mingw-w64 straight up. No one before you has seriously worked with that stuff in years.

cctools asserts targetPlatform.isDarwin, which fails.

This should be correct / not affect you targeting windows? Are you working on mac -> windows too?

ghc binary expressions assert stdenv.targetPlatform == stdenv.hostPlatform, which fails during validation.

This is intended. The binary ghcs are not cross compilers.

The buildMK in the ghc expressions is wrong and needs to also include the relevant $flavour.mk

Oooo thanks. You obviously know more about this than me, so glad you could be here to catch my mistake :).

The generic builder uses hsc2hs from the nativeGhc, but should use $targetPrefix-hsc2hs.

👍 Yeah this was some last-minute hack done be somebody a year ago that was never revisited.

haskellSetupDepends should come from the buildPackages package set as they are compiled and run by the native compiler.

I'd be loath to add more callPackage-based splicing, so it would be nice to just have cabal2nix get these from buildHaskellPackages.

[ ] packageOverrides and buildPackages don't play nice together. Injecting a custom ghc via config doesn't work anymore, as buildPackages is fixed prior to the overrides being applied, and buildPackages doesn't see any added packages from the packageOverrides.

Oh hmm? This is the opposite of the conclusion I came to talking to you on IRC. Did you confirm this after the fact?

@angerman
Copy link
Contributor Author

angerman commented Mar 6, 2018

@Ericson2314, I'll hopefully have some clean up ready soon.

This should be correct / not affect you targeting windows? Are you working on mac -> windows too?

It's not. And yes I do mac -> windows, and hope linux -> windows just falls out as nixpkgs (linux) support is usually much better than nixpkgs (macos).

ghc binary expressions assert stdenv.targetPlatform == stdenv.hostPlatform, which fails during validation.

This is intended. The binary ghcs are not cross compilers.

Correct, they are not cross compilers. But they are used to build the cross compilers. E.g. I use the 8.2 binary for bootstrapping the cross compiler. And hence I run into that assert.

haskellSetupDepends should come from the buildPackages package set as they are compiled and run by the native compiler.

I'd be loath to add more callPackage-based splicing, so it would be nice to just have cabal2nix get these from buildHaskellPackages.

Right. cabal2nix will be an additional can of worms. (E.g. configuring the packages for the target properly... -- if I want to build windows, I need os(windows) to properly fire. cabal2nix tries to compute this statically; but there is some logic for windows and macOS already I believe; we might
need some additional logic for cross compilation though).

packageOverrides and buildPackages don't play nice together. Injecting a custom ghc via config doesn't work anymore, as buildPackages is fixed prior to the overrides being applied, and buildPackages doesn't see any added packages from the packageOverrides.

Oh hmm? This is the opposite of the conclusion I came to talking to you on IRC. Did you confirm this after the fact?

I think I did. I believe the core issue is the order in which the buildPackages and overrides are applied. At least the approach I've taken in https://github.com/input-output-hk/nix-hs-hello-windows/blob/d790960552792d5daa250594e17371f0fef261e8/config.nix, required to use customizeGhc on both the packages and the buildPackages. Just adding a compiler to the package.haskell.compiler set, does not result in that compiler showing up in the buildPackages.compiler set.

@Ericson2314
Copy link
Member

Ericson2314 commented Mar 8, 2018

@angerman sorry to not be getting back sooner.

And yes I do mac -> windows

Cool. @LnL7 and I were working on on mac -> linux things that may help insofar that they fix general infra issues. I'll be working on some ios stuff shortly for work, which may also help.

But they are used to build the cross compilers... And hence I run into that assert.

But then they should come from a previous bootstrapping stage (some .buildPackages) such that the assert isn't hit. Those variables are per-stage.

We might need some additional logic for cross compilation though

Yeah there's quite likely some low hanging fruit to make it work for both without adding some complexity. Just a matter of using the right idioms.

I think I did.

I'm going to check this right now. edit https://github.com/input-output-hk/nix-hs-hello-windows/pull/2/files

@angerman
Copy link
Contributor Author

angerman commented Mar 9, 2018

@Ericson2314

  • for the iOS stuff you may want to use the llvmng backend; ping me when you are building the compiler.
  • the derivation reuses the ghcHEAD attribute, I could not make it work injecting a custom new attribute (e.g. myghc) as that didn’t show up in the buildPackags.

@Ericson2314
Copy link
Member

Ericson2314 commented Mar 9, 2018

Will ping you! Added the custom attribute.

@nomeata
Copy link
Contributor

nomeata commented Mar 6, 2019

What is the state of this? Is this related to the cross compilation support that I can get (in theory) via

nix-build -A pkgsCross.mingw32.haskellPackages.hello

or is that separate endevour?

(I’d be very keen on using nix and cross compilations to make the Windows releases for tttool, my probably most impactful Haskell program so far.)

@nomeata
Copy link
Contributor

nomeata commented Mar 6, 2019

JFTR, this currently (b1f767e) fails building GHC, after running for quite some time:

"inplace/bin/ghc-stage1" -optc-fno-stack-protector -optc-Wall -optc-Wall -optc-Wextra -optc-Wstrict-prototypes -optc-Wmissing-prototypes -optc-Wmissing-declarations -optc-Winline -optc-Waggregate-return -optc-Wpointer-arith -optc-Wmissing-noreturn -optc-Wnested-externs -optc-Wredundant-decls -optc-Wundef -optc-Iincludes -optc-Iincludes/dist -optc-Iincludes/dist-derivedconstants/header -optc-Iincludes/dist-ghcconstants/header -optc-Irts -optc-Irts/dist/build -optc-DCOMPILING_RTS -optc-DFS_NAMESPACE=rts -optc-fno-strict-aliasing -optc-fno-common -optc-Irts/dist/build/./autogen -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -optc-O2 -optc-fomit-frame-pointer -optc-g -optc-DRtsWay=\"rts_thr_p\" -optc-DWINVER=0x06000100 -static -prof -eventlog -optc-DTHREADED_RTS  -O -H64m -Wall   -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -Irts -Irts/dist/build -DCOMPILING_RTS -DFS_NAMESPACE=rts -this-unit-id rts -dcmm-lint      -i -irts -irts/dist/build -Irts/dist/build -irts/dist/build/./autogen -Irts/dist/build/./autogen            -O2 -fPIC -Wcpp-undef    -Wnoncanonical-monad-instances  -c rts/sm/GC.c -o rts/dist/build/sm/GC.thr_p_o

<no location info>: error:
    Warning: Couldn't figure out LLVM version!
             Make sure you have installed LLVM 6.0

<no location info>: error:
    Warning: Couldn't figure out LLVM version!
             Make sure you have installed LLVM 6.0
ghc-stage1: could not execute: opt
make[1]: *** [libraries/ghc-prim/ghc.mk:4: libraries/ghc-prim/dist-install/build/GHC/Types.o] Error 1
make[1]: *** Waiting for unfinished jobs....
ghc-stage1: could not execute: opt
make[1]: *** [libraries/ghc-prim/ghc.mk:4: libraries/ghc-prim/dist-install/build/GHC/Types.p_o] Error 1
make: *** [Makefile:127: all] Error 2
builder for '/nix/store/1grlmks733ryhm1ffkf360v11v0bzx05-i686-pc-mingw32-ghc-8.6.4.drv' failed with exit code 2
cannot build derivation '/nix/store/g8qnhgzw8x6gflnzyiljf24vdyg7dfrk-tttool-1.8-i686-pc-mingw32.drv': 1 dependencies couldn't be built
error: build of '/nix/store/g8qnhgzw8x6gflnzyiljf24vdyg7dfrk-tttool-1.8-i686-pc-mingw32.drv' failed

@angerman
Copy link
Contributor Author

angerman commented Mar 7, 2019

@nomeata We use https://github.com/input-output-hk/haskell.nix for our cross compilation (to windows) needs at IOHK.

@nomeata
Copy link
Contributor

nomeata commented Mar 16, 2019

So which of the patches https://github.com/input-output-hk/iohk-nix/tree/master/patches/ghc do we need to get into nixpkgs to get

nix-build -A pkgs.pkgsCross.mingwW64.haskellPackages.hello

to compile?

@albertov
Copy link

I've just discovered https://github.com/input-output-hk/haskell.nix thanks to this issue and I feel like a kid in a candy store. This is very exciting! @angerman is TemplateHaskell supported when cross-compiling with this tooling?

@nomeata
Copy link
Contributor

nomeata commented Mar 17, 2019

With this patch, I can actually build the GHC minwW64-targeting GHC:

diff --git a/pkgs/development/compilers/ghc/8.6.4.nix b/pkgs/development/compilers/ghc/8.6.4.nix
index 140cea22442..b0dacae5e2a 100644
--- a/pkgs/development/compilers/ghc/8.6.4.nix
+++ b/pkgs/development/compilers/ghc/8.6.4.nix
@@ -31,7 +31,11 @@
 
 , # What flavour to build. An empty string indicates no
   # specific flavour and falls back to ghc default values.
-  ghcFlavour ? stdenv.lib.optionalString (stdenv.targetPlatform != stdenv.hostPlatform) "perf-cross"
+  ghcFlavour ?
+    stdenv.lib.optionalString
+      (stdenv.targetPlatform != stdenv.hostPlatform)
+      (if stdenv.targetPlatform.isWindows then "perf-cross-ncg" else "perf-cross")
+
 }:
 
 assert !enableIntegerSimple -> gmp != null;

Not sure how good it is, though. I can build hello, but can’t run it immediately :

~/build/nixpkgs $ WINEPREFIX=~/.wine64 wine64 $(nix-build  -A pkgsCross.mingwW64.haskell.packages.ghc864.hello)/bin/hello.exe
0009:err:module:import_dll Library libffi-6.dll (which is needed by L"Z:\\nix\\store\\ngick88q873hgdjzml0arb7nylycdwb8-hello-1.0.0.2-x86_64-pc-mingw32\\bin\\hello.exe") not found
0009:err:module:attach_dlls Importing dlls for L"Z:\\nix\\store\\ngick88q873hgdjzml0arb7nylycdwb8-hello-1.0.0.2-x86_64-pc-mingw32\\bin\\hello.exe" failed, status c0000135

But packages that invoke haddock fail to build:

~/build/nixpkgs $ nix-build  -A pkgsCross.mingwW64.haskell.packages.ghc864.generics-sop
…
Building library for sop-core-0.4.0.0..
[1 of 8] Compiling Data.SOP.BasicFunctors ( src/Data/SOP/BasicFunctors.hs, dist/build/Data/SOP/BasicFunctors.o )
[2 of 8] Compiling Data.SOP.Constraint ( src/Data/SOP/Constraint.hs, dist/build/Data/SOP/Constraint.o )
[3 of 8] Compiling Data.SOP.Classes ( src/Data/SOP/Classes.hs, dist/build/Data/SOP/Classes.o )
[4 of 8] Compiling Data.SOP.Sing    ( src/Data/SOP/Sing.hs, dist/build/Data/SOP/Sing.o )
[5 of 8] Compiling Data.SOP.NP      ( src/Data/SOP/NP.hs, dist/build/Data/SOP/NP.o )
[6 of 8] Compiling Data.SOP.NS      ( src/Data/SOP/NS.hs, dist/build/Data/SOP/NS.o )
[7 of 8] Compiling Data.SOP.Dict    ( src/Data/SOP/Dict.hs, dist/build/Data/SOP/Dict.o )
[8 of 8] Compiling Data.SOP         ( src/Data/SOP.hs, dist/build/Data/SOP.o )
[1 of 8] Compiling Data.SOP.BasicFunctors ( src/Data/SOP/BasicFunctors.hs, dist/build/Data/SOP/BasicFunctors.p_o )
[2 of 8] Compiling Data.SOP.Constraint ( src/Data/SOP/Constraint.hs, dist/build/Data/SOP/Constraint.p_o )
[3 of 8] Compiling Data.SOP.Classes ( src/Data/SOP/Classes.hs, dist/build/Data/SOP/Classes.p_o )
[4 of 8] Compiling Data.SOP.Sing    ( src/Data/SOP/Sing.hs, dist/build/Data/SOP/Sing.p_o )
[5 of 8] Compiling Data.SOP.NP      ( src/Data/SOP/NP.hs, dist/build/Data/SOP/NP.p_o )
[6 of 8] Compiling Data.SOP.NS      ( src/Data/SOP/NS.hs, dist/build/Data/SOP/NS.p_o )
[7 of 8] Compiling Data.SOP.Dict    ( src/Data/SOP/Dict.hs, dist/build/Data/SOP/Dict.p_o )
[8 of 8] Compiling Data.SOP         ( src/Data/SOP.hs, dist/build/Data/SOP.p_o )
haddockPhase
Preprocessing library for sop-core-0.4.0.0..
Running Haddock on library for sop-core-0.4.0.0..
Warning: The documentation for the following packages are not installed. No
links will be generated to these packages: array-0.5.3.0, base-4.12.0.0,
deepseq-1.4.4.0, ghc-prim-0.5.3, integer-gmp-1.0.2.0
Warning: --source-* options are ignored when --hyperlinked-source is enabled.
Haddock coverage:
Bad interface file: /nix/store/3v69m9hlakn5iiz2jv5zm8hdxyx4d0pi-x86_64-pc-mingw32-ghc-8.6.4/lib/x86_64-pc-mingw32-ghc-8.6.4/base-4.12.0.0/GHC/Base.hi
    haddock: panic! (the 'impossible' happened)
  (GHC version 8.6.4 for x86_64-unknown-linux):
        getSymtabName:unknown known-key unique

2386560102
(9, 1126)
Call stack:
    CallStack (from HasCallStack):
      callStackDoc, called at compiler/utils/Outputable.hs:1160:37 in ghc:Outputable
      pprPanic, called at compiler/iface/BinIface.hs:379:34 in ghc:BinIface

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

builder for '/nix/store/bgz26jbkwzymfhlnyzbip3mcrzasd6jv-sop-core-0.4.0.0-x86_64-pc-mingw32.drv' failed with exit code 1
error: build of '/nix/store/bgz26jbkwzymfhlnyzbip3mcrzasd6jv-sop-core-0.4.0.0-x86_64-pc-mingw32.drv' on 'ssh://[email protected]' failed: builder for '/nix/store/bgz26jbkwzymfhlnyzbip3mcrzasd6jv-sop-core-0.4.0.0-x86_64-pc-mingw32.drv' failed with exit code 1
builder for '/nix/store/bgz26jbkwzymfhlnyzbip3mcrzasd6jv-sop-core-0.4.0.0-x86_64-pc-mingw32.drv' failed with exit code 1
cannot build derivation '/nix/store/8zpf673zprrjgicm07yhxs8w9m1zq4z2-generics-sop-0.4.0.1-x86_64-pc-mingw32.drv': 1 dependencies couldn't be built
error: build of '/nix/store/8zpf673zprrjgicm07yhxs8w9m1zq4z2-generics-sop-0.4.0.1-x86_64-pc-mingw32.drv' failed

@angerman
Copy link
Contributor Author

angerman commented Mar 17, 2019

@albertov: yes it does support TH if you can construct a runner. Our windows cross compilation infrastructure uses wine for this. grep for withTH in input-output-hk/iohk-nix.

As a usage example check out input-output-hk/cardano-chain (hint, also check the relevant builds on our hydra instance to see the jobs we define).

@nomeata using cross-ncg uses the native code gen instead of the llvm one for cross compilation. This is only available when targeting x86_64.

There are sadly quite a few patches involved. The intention is to upstream them all though.

The library issue with windows in nix is mildly annoying. As a general hack you can either try to build as much statically as possible, or collect all .dlls from the dependency closure and copy them over into $out/bin.

As a general note: I have absolutely no idea how well the haskell packages in nix are suited for windows cross compilation. They are I believe generated with cabal2nix and as such do not contain any conditionals from the cabal files. They are flattened at generation time and most likely to os=linux and some predefined flags :-(

@angerman
Copy link
Contributor Author

One more note: I do trust the ncg more than the llvm backend. The ncg is the default backend for almost every ghc build you come across.

@stale
Copy link

stale bot commented Jun 3, 2020

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

  1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
  2. Ask on the NixOS Discourse.
  3. Ask on the #nixos channel on irc.freenode.net.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 3, 2020
@nomeata
Copy link
Contributor

nomeata commented Apr 6, 2021

Maybe the title of this PR should include “Windows”? :)

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 6, 2021
@nomeata
Copy link
Contributor

nomeata commented Apr 6, 2021

Just gave it a shot with current master, trying to use 8.8.4 from nixpkgs.pkgsCross.mingwW64. I first got stuck with haskell/cabal#5887 which was fixed, it seems, using https://raw.githubusercontent.com/input-output-hk/haskell.nix/11ac089c9ad436defc77fd096e00bebc42adda1d/overlays/patches/ghc/cabal-host.patch. Now I get

/nix/store/ix7ig152gf5dxhb9fhfk3v6q4vc3vawb-x86_64-w64-mingw32-binutils-2.31.1/bin/x86_64-w64-mingw32-ld: libraries/base/dist-install/build/HSbase-4.13.0.0.p_o: too many sections (54972)
/nix/store/ix7ig152gf5dxhb9fhfk3v6q4vc3vawb-x86_64-w64-mingw32-binutils-2.31.1/bin/x86_64-w64-mingw32-ld: final link failed: file too big
make[1]: *** [libraries/base/ghc.mk:4: libraries/base/dist-install/build/HSbase-4.13.0.0.p_o] Error 1

Probably also just a patch I have to get from somehere… but lunch first.

@angerman
Copy link
Contributor Author

angerman commented Apr 7, 2021

/nix/store/ix7ig152gf5dxhb9fhfk3v6q4vc3vawb-x86_64-w64-mingw32-binutils-2.31.1/bin/x86_64-w64-mingw32-ld: libraries/base/dist-install/build/HSbase-4.13.0.0.p_o: too many sections (54972)

So the profiling object has too many sections. The best solution for this I could come up with is to disable profiling.

@Ericson2314 Ericson2314 changed the title Cross Compiling Haskell Cross Compiling Haskell to Windows Apr 7, 2021
@nomeata
Copy link
Contributor

nomeata commented Apr 7, 2021

JFTR: After including all of the GHC patches from haskell.nix I got past that stage.

@jappeace
Copy link
Contributor

is there any progress on this? Having nix cross compile a haskell program would save me a lot of tedious windows based building.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: cross-compilation Building packages on a different platform than they will be used on 6.topic: haskell 6.topic: windows Running, or buiding, packages on Windows
Projects
None yet
Development

No branches or pull requests

5 participants