Replace nixpkgs_cc_configure() with a toolchain

[aherrmann]

Depends on #132
Closes #88

This introduces a new repository rule nixpkgs_cc_configure_hermetic which is meant to replace nixpkgs_cc_configure. The latter is implemented by calling Bazel's builtin CC toolchain autodetection with the binary paths overriden to point to Nix provided tools. This is more hermetic than Bazel's default behavior in that the tools are not just picked from $PATH. However, it is not hermetic. Bazel's CC toolchain autodetection is influenced by environment variables, and it shells out to the provided cc binary in the context of a repository rule to determine system include paths which is also affected by environment variables. Concretely, this caused cache misses due to include paths provided by a nix-shell changing the cache keys of build actions depending on the CC toolchain.

nixpkgs_cc_configure_hermetic on the other hand uses binaries determined by a Nix derivation and performs additional configuration such as determining system include paths within a Nix build, i.e. within Nix's own sandbox. The results are stored in a text file CC_TOOLCHAIN_INFO which is then parsed in a Bazel repository rule. From there on the toolchain configuration follows the example of Bazel's builtin CC toolchain for Unix: https://github.com/bazelbuild/bazel/blob/0f4c498a270f05b3896d57055b6489e824821eda/tools/cpp/unix_cc_configure.bzl#L310.

Users can override the Nix provided compiler using the nix_file or nix_file_content and attribute_path arguments. The corresponding Nix derivation should produce the required tools, e.g. bin/cc. The implementation of this feature required a way to pass the path to a Nix file (the nix_file argument) to the nixopts attribute of nixpkgs_package from within a macro. This is difficult to do correctly without repository_ctx available. Instead, of trying to predict this path from within a macro a more robust approach is to define location expansion for the nixopts attribute using $(location ...) patterns similar to ctx.expand_location. This feature is implemented in #132.

[aherrmann]

To be consistent with Bazel's builtin location expansion this should error on unescaped instances of $() that are not $(location ...).

[guibou]

This is a huge change, LGTM with a diagonal read and it does not break anything for me. But take this with a grain of salt ;)

This being said, nixpkgs_cc_configure may still be necessary in some context. On a client codebase, we are using a toolchain (with a setup similar to the one involved here) but still observes that bazel is still looking for a system toolchain for some tasks (this is a bootstrap problem).

So I do think that nixpkgs_cc_configure_hermetic is not a better nixpkgs_cc_configure, just that it solves a different problem.

Side question, can this toolchain configuration made more composable so people who are already setting up a CC toolchain may just inherit part of it so their CC toolchain will become Haskell compatible?

[aherrmann]

This being said, nixpkgs_cc_configure may still be necessary in some context. On a client codebase, we are using a toolchain (with a setup similar to the one involved here) but still observes that bazel is still looking for a system toolchain for some tasks (this is a bootstrap problem).

In cases where Bazel looks at local_config_cc, e.g. because it is hard-coded someplace, you can use nixpkgs_cc_configure_hermetic(name = "local_config_cc", ...). This overrides the builtin CC toolchain just as nixpkgs_cc_configure would, but is more hermetic.

[aherrmann]

Side question, can this toolchain configuration made more composable so people who are already setting up a CC toolchain may just inherit part of it so their CC toolchain will become Haskell compatible?

Can you describe the use-case you have in mind? nixpkgs_configure_cc_hermetic allows users to provide their own Nix expression, so it should be very flexible on that side.

Note, this toolchain is not a prerequisite for rules_haskell. E.g. with rules_haskell with GHC from bindist you can just use Bazel's builtin CC toolchain. It's just that a nixpkgs provided GHC and Bazel's builtin CC toolchain don't work well together. However, if you have another means to provide a CC toolchain that works with a nixpkgs GHC, then you don't have to use this one.

[aherrmann]

@micahcc Thank you for testing this! Indeed, I can reproduce the issue. The MacOS version of the toolchain fixes this with additional flags using makeWrapper. Though there seems to be a difference between MacOS and Linux regarding linker flags.

Unfortunately, always using c++ instead of cc would break compilation of regular C code that happens to be invalid C++. However, the Nix clang wrapper picks up the -x c++ flag. So, adding that on C++ compilation avoids the missing include paths. I've pushed the necessary changes here. Does that fix the issue for your use-case?

[aherrmann]

I've been able to reproduce the clang issue on Linux with the previous nixpkgs_cc_configure implementation as well. Meaning, this is not a regression introduced by this change. In that case one has to point the envvar CC to the Nix provided clang since the builtin autoconfigure cc toolchain otherwise uses gcc on Linux.

To move forward with this I'll merge this PR and open a separte issue and PR for the clang issue.

[mboes]

@aherrmann is there a ticket tracking this? Once this flag is enabled, do I understand right that --crosstool_top will no longer be necessary?

[aherrmann]

@aherrmann is there a ticket tracking this?

I've looked into the issue in the daml repo once more. It looks like this is a rules_haskell issue: tweag/rules_haskell#1467

However, in general bazelbuild/bazel#1258 suggests that there is still ongoing work in Bazel in this area. And bazelbuild/bazel#12712 also seems related.

Once this flag is enabled, do I understand right that --crosstool_top will no longer be necessary?

Correct, assuming that all involved Starlark code is compatible with cc toolchain resolution. E.g. any direct reference to ctx.attr._cc_toolchain breaks this and causes a dependency on --crosstool_top.