Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial WebAssembly/WASI cross-compilation support #56555

Merged
merged 12 commits into from
Apr 24, 2019

Conversation

matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Mar 1, 2019

Motivation for this change

Need to add LLVM 8 to get this working. Can compile basic C programs. Things like C++ are unfortunately not supported right now.

/cc @ElvishJerricco

Test case:

with import <nixpkgs> {
  crossSystem = (import <nixpkgs/lib>).systems.examples.wasi32;
};
let
  cpp = builtins.toFile "test.c++" ''
    #include <iostream>

    int main() {
      std::cout << "Hello world!" << std::endl;
    }
  '';
in runCommandCC "cross-test" {} ''
      $CXX -o test ${cpp}
      ${stdenv.hostPlatform.emulator buildPackages} test > $out
''
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@GrahamcOfBorg GrahamcOfBorg added the 6.topic: stdenv Standard environment label Mar 1, 2019
Copy link
Contributor

@ElvishJerricco ElvishJerricco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just took a cursory glance. Will do a more in depth review tomorrow.

In general though, this looks like several PRs in one. Stuff like the new version of llvm and the V8 changes seem like they should be different PRs to get better review.

pkgs/build-support/cc-wrapper/default.nix Outdated Show resolved Hide resolved
pkgs/development/compilers/llvm/7/compiler-rt.nix Outdated Show resolved Hide resolved
pkgs/os-specific/linux/musl/web.nix Outdated Show resolved Hide resolved
@matthewbauer
Copy link
Member Author

Yeah good point, i can split into llvm8 init, wasm init, and the v8 update.

lib/systems/parse.nix Outdated Show resolved Hide resolved
@qknight
Copy link
Member

qknight commented Mar 4, 2019

this looks really cool!

@matthewbauer matthewbauer changed the title Initial WebAssembly cross-compilation support [wip] Initial WebAssembly cross-compilation support Mar 9, 2019
@matthewbauer matthewbauer changed the title [wip] Initial WebAssembly cross-compilation support Initial WebAssembly cross-compilation support Mar 22, 2019
@ElvishJerricco
Copy link
Contributor

This should probably switch to https://github.com/CraneStation/wasi-sysroot for libc. WASI is likely to become pretty standard, unlike both the jfbastien and the WebGHC ones.

@matthewbauer
Copy link
Member Author

@ElvishJerricco Yeah i agree on that. Ideally the jfbastien and webghc musl stuff could be upstreamed. But right now that's not really a real ABI. Luckily we can reuse most of the changes here though.

@matthewbauer matthewbauer changed the title Initial WebAssembly cross-compilation support Initial WebAssembly/WASI cross-compilation support Apr 13, 2019
@matthewbauer matthewbauer changed the base branch from staging to master April 24, 2019 02:03
@matthewbauer matthewbauer changed the base branch from master to staging April 24, 2019 02:03
@matthewbauer
Copy link
Member Author

@GrahamcOfBorg eval

@ofborg ofborg bot removed 6.topic: GNOME GNOME desktop environment and its underlying platform 6.topic: haskell 6.topic: lua 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: printing 6.topic: python 6.topic: qt/kde 6.topic: rust 6.topic: steam 6.topic: vim 6.topic: xfce The Xfce Desktop Environment 8.has: documentation 8.has: module (update) This PR changes an existing module in `nixos/` 2.status: merge conflict This PR has merge conflicts with the target branch labels Apr 24, 2019
@matthewbauer matthewbauer merged commit 7488a36 into NixOS:staging Apr 24, 2019
@dtzWill
Copy link
Member

dtzWill commented Apr 24, 2019 via email

iohk-bors bot added a commit to cardano-foundation/cardano-wallet that referenced this pull request Jan 29, 2020
1265: Update nixpkgs channel to 19.09 r=rvl a=angerman

# Overview

This updates the nixpkgs version to 19.09 (latest stable release) from 19.03 (unsupported old release).

# Details

haskell.nix (as referenced in this repository) for the purpose of running CI, was using a custom nixpkgs-19.03 branch. It has been the goal for a while to upgrade from nixpkgs-19.03 to nixpkgs-19.09. Ideally to the stock upstream one without custom patches. This PR updates the nixpkgs-19.03 pin to nixpkgs-19.09 and fixes the issues (as encountered below) in haskell.nix. Subsequently this PR also contains a updated haskell.nix pin. The `w64` issue stems from nixpkgs changing the identifier for windows cross from `x86_64-pc-mingw32` to `x86_64-w64-mingw32` from nixpkgs 19.03 to 19.09. This interacts badly with the jormungandr reference for windows, and @rvl will take a look at that within cardano-wallet, as it can not be fixed in haskell.nix.

# Comments

Issues encountered:
- [x] ```error: attribute 'series3' missing, at /nix/store/87zygi9fxvgqkdflsd5hmx947aj3mf1m-nixpkgs/pkgs/misc/emulators/wine/packages.nix:6:11```
     This is due to missing NixOS/nixpkgs#71216

- [x] ```Unknown vendor: w64```
   This seems to be due to jourmungandr using some 18.09git checkout, which doesn't sport the `w64` vendor, but uses `pc` instead.

- [x] ```/nix/store/d7q2kp5saj7xd9wii7cyfis8y3s4d2sj-network-2.6.3.6-setup-x86_64-w64-mingw32/bin/Setup: error while loading shared libraries: libgmp.so.10: cannot open shared object file: No such file or directory```
   Which is due to
   ```
   ldd /nix/store/d7q2kp5saj7xd9wii7cyfis8y3s4d2sj-network-2.6.3.6-setup-x86_64-w64-mingw32/bin/Setup
           linux-vdso.so.1 (0x00007ffeb9369000)
           libm.so.6 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/libm.so.6 (0x00007f7a4695c000)
           librt.so.1 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/librt.so.1 (0x00007f7a46952000)
           libutil.so.1 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/libutil.so.1 (0x00007f7a4694d000)
           libdl.so.2 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/libdl.so.2 (0x00007f7a46948000)
           libpthread.so.0 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/libpthread.so.0 (0x00007f7a46927000)
           libgmp.so.10 => not found
           libffi.so.6 => not found
           libc.so.6 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/libc.so.6 (0x00007f7a4676f000)
           /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/ld-linux-x86-64.so.2 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007f7a46af4000)
   ```
   Which originated from NixOS/nixpkgs#56555 where `NIX_DONT_SET_RPATH` is now set on more leniently.  This requires us to properly fix the build level we build the Setup.hs with in haskell.nix; it needs to come from the `buildPackages.stdenv`, not from the `stdenv`, which in the case of cross compilation would be the one for the final host.


[Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-1265)


Co-authored-by: Moritz Angermann <[email protected]>
Co-authored-by: Rodney Lorrimar <[email protected]>
iohk-bors bot added a commit to cardano-foundation/cardano-wallet that referenced this pull request Jan 30, 2020
1265: Update nixpkgs channel to 19.09 r=rvl a=angerman

# Overview

This updates the nixpkgs version to 19.09 (latest stable release) from 19.03 (unsupported old release).

# Details

haskell.nix (as referenced in this repository) for the purpose of running CI, was using a custom nixpkgs-19.03 branch. It has been the goal for a while to upgrade from nixpkgs-19.03 to nixpkgs-19.09. Ideally to the stock upstream one without custom patches. This PR updates the nixpkgs-19.03 pin to nixpkgs-19.09 and fixes the issues (as encountered below) in haskell.nix. Subsequently this PR also contains a updated haskell.nix pin. The `w64` issue stems from nixpkgs changing the identifier for windows cross from `x86_64-pc-mingw32` to `x86_64-w64-mingw32` from nixpkgs 19.03 to 19.09. This interacts badly with the jormungandr reference for windows, and @rvl will take a look at that within cardano-wallet, as it can not be fixed in haskell.nix.

# Comments

Issues encountered:
- [x] ```error: attribute 'series3' missing, at /nix/store/87zygi9fxvgqkdflsd5hmx947aj3mf1m-nixpkgs/pkgs/misc/emulators/wine/packages.nix:6:11```
     This is due to missing NixOS/nixpkgs#71216

- [x] ```Unknown vendor: w64```
   This seems to be due to jourmungandr using some 18.09git checkout, which doesn't sport the `w64` vendor, but uses `pc` instead.

- [x] ```/nix/store/d7q2kp5saj7xd9wii7cyfis8y3s4d2sj-network-2.6.3.6-setup-x86_64-w64-mingw32/bin/Setup: error while loading shared libraries: libgmp.so.10: cannot open shared object file: No such file or directory```
   Which is due to
   ```
   ldd /nix/store/d7q2kp5saj7xd9wii7cyfis8y3s4d2sj-network-2.6.3.6-setup-x86_64-w64-mingw32/bin/Setup
           linux-vdso.so.1 (0x00007ffeb9369000)
           libm.so.6 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/libm.so.6 (0x00007f7a4695c000)
           librt.so.1 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/librt.so.1 (0x00007f7a46952000)
           libutil.so.1 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/libutil.so.1 (0x00007f7a4694d000)
           libdl.so.2 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/libdl.so.2 (0x00007f7a46948000)
           libpthread.so.0 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/libpthread.so.0 (0x00007f7a46927000)
           libgmp.so.10 => not found
           libffi.so.6 => not found
           libc.so.6 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib/libc.so.6 (0x00007f7a4676f000)
           /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/ld-linux-x86-64.so.2 => /nix/store/pnd2kl27sag76h23wa5kl95a76n3k9i3-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007f7a46af4000)
   ```
   Which originated from NixOS/nixpkgs#56555 where `NIX_DONT_SET_RPATH` is now set on more leniently.  This requires us to properly fix the build level we build the Setup.hs with in haskell.nix; it needs to come from the `buildPackages.stdenv`, not from the `stdenv`, which in the case of cross compilation would be the one for the final host.


[Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-1265)


Co-authored-by: Moritz Angermann <[email protected]>
Co-authored-by: Rodney Lorrimar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants