System API for ECDSA signing

[marcin-dziadus]

This PR Implements a corresponding part of the IC spec: dfinity/interface-spec#6

It also splits up IC.Test.Agent and IC.Test.Spec to reduce the excessive memory consumption of compiling those modules.

[basvandijk]

LGTM!

I'll help with fixing CI.

[basvandijk]

Status report

For building ic-ref-dist I switched to pkgsStatic to get a static secp256k1. However pkgsStatic removes all dynamic libraries (.so). This is problematic for Haskell modules using TemplateHaskell because they require dynamic libraries to be available.

See: NixOS/nixpkgs#61575

For this reason I had to remove TemplateHaskell support from QuickCheck. But haskell-candid also uses TemplateHaskell which can't be disabled. So that package is failing now.

Not yet sure how to work around this...

[basvandijk]

The TemplateHaskell issues have been fixed by patching GHC and disabling haddock in a few libraries.

Now hopefully the last remaining issue is the following error when building ic-ref-dist:

src/IC/Crypto/BLS.hsc:23 directive strict_import cannot be handled in cross-compilation mode

@nomeata mentioned a similar issue in rethab/bindings-dsl#38.

[nomeata]

So maybe 47e1274 cab be cherry-picked?

[basvandijk]

I think IC.Crypto.BLS is simple enough to replace bindings-DSL with raw hsc.

[basvandijk]

So maybe 47e1274 cab be cherry-picked?

Done. @nomeata thanks for the pointer!

[basvandijk]

Next problem is that ic-ref-dist refers to the warp Haskell library which is not allowed (allowedReferences= []).

The reason is that the ic-ref executable from ic-hs-static references warp:

$ grep -a /nix/store/rg01ag16m4kk1xrp6ypw6ypj0wqy5bmk-warp-static-x86_64-unknown-linux-musl-3.3.17 \
          /nix/store/27samy25mh98r4glq2krkmv68l9mxb5a-ic-hs-static-x86_64-unknown-linux-musl-0.0.1/bin/ic-ref
...warp_bindir/nix/store/rg01ag16m4kk1xrp6ypw6ypj0wqy5bmk-warp-static-x86_64-unknown-linux-musl-3.3.17/binwarp_libdir/nix/store/rg01ag16m4kk1xrp6ypw6ypj0wqy5bmk-warp-static-x86_64-unknown-linux-musl-3.3.17/lib/ghc-8.10.7/x86_64-linux-ghc-8.10.7/warp-3.3.17-LFuiV3JNZfpKQMWWUSmbjdwarp_dynlibdir/nix/store/rg01ag16m4kk1xrp6ypw6ypj0wqy5bmk-warp-static-x86_64-unknown-linux-musl-3.3.17/lib/ghc-8.10.7/x86_64-linux-ghc-8.10.7warp_datadir/nix/store/rg01ag16m4kk1xrp6ypw6ypj0wqy5bmk-warp-static-x86_64-unknown-linux-musl-3.3.17/share/x86_64-linux-ghc-8.10.7/warp-3.3.17warp_libexecdir/nix/store/rg01ag16m4kk1xrp6ypw6ypj0wqy5bmk-warp-static-x86_64-unknown-linux-musl-3.3.17/libexec/x86_64-linux-ghc-8.10.7/warp-3.3.17warp_sysconfdir/nix/store/rg01ag16m4kk1xrp6ypw6ypj0wqy5bmk-warp-static-x86_64-unknown-linux-musl-3.3.17/etc...

Note that the references to warp are the only references to the /nix/store.

[basvandijk]

@nomeata any idea how these paths to warp end up in ic-ref?

[basvandijk]

Just linking NixOS/nixpkgs#43795 (comment) here because I think we can simplify our static linking setup when we upgrade to release-22.05 (like removing our GHC override).

[nomeata]

Next problem is that ic-ref-dist refers to the warp Haskell library which is not allowed (allowedReferences= []).

If these paths are actually unused and don’t affect whether it runs, you can use removeReferencesTo to be unblocked here, right?

I guess they are there because warp uses the Paths_warp module generated by Cabal, but only use the version number, so probably fine to remove:

/tmp/warp-3.3.19 $ grep -r -w Paths_warp 
warp.cabal:                     Paths_warp
warp.cabal:                     Paths_warp
Network/Wai/Handler/Warp/Response.hs:import qualified Paths_warp
Network/Wai/Handler/Warp/Response.hs:warpVersion = showVersion Paths_warp.version
Network/Wai/Handler/Warp/Settings.hs:import qualified Paths_warp
Network/Wai/Handler/Warp/Settings.hs:    , settingsServerName = C8.pack $ "Warp/" ++ showVersion Paths_warp.version

[basvandijk]

I guess they are there because warp uses the Paths_warp

On our way to the cinema to see Top Gun my brother @roelvandijk and I came to the same conclusion when discussing this problem ;)

Thanks for confirming our hypothesis!

I'll remove the references with removeReferencesTo.