-
-
Notifications
You must be signed in to change notification settings - Fork 14.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP/RFC] Make cmakeFlags
more ergonomic
#17886
Conversation
@aneeshusa, thanks for your PR! By analyzing the annotation information on this pull request, we identified @christopherpoole, @jraygauthier and @vcunat to be potential reviewers |
86bd73d
to
b64b29d
Compare
cmakeFlags = [ | ||
"-DSC_WII=OFF" | ||
"-DSC_EL=${if useSCEL then "ON" else "OFF"}" | ||
]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not this?
cmakeFlags = {
SC_WII = false;
SC_EL = useSCEL;
};
@rasendubi there are over 300 files that use |
I think cmakeFlags should be a list, not a set. A set feels like wrong abstraction to me: a command-line interface is a list of arguments, not a set. (How to specify "foo -v -v" (double verbose) if args is a set?). Just my opinion. |
@bjornfor The set is for the FWIW, I find the |
@bjornfor Almost all uses of |
|
@vcunat This is meant to simplify the 95% case, while still making the 5% case possible. I asked on #cmake and for In general, I would like to see build tools move towards passing more structured data (like attribute sets) between themselves, and view the form in which they are passed (environment variables, command line flags) just as a serialization detail. I think Nix and nixpkgs can help with this by making structured data more available during building (attrsets, lists, JSON), hence the PR. Supporting both styles will be hard to do. This change already breaks Also, I'd like feedback on |
Speaking of myself, I just understood that I've thought and used I'm now confused why that worked anyway. In all my usecases |
@aneeshusa, @gebner: Ok, so a "set" is a nice abstraction for a part of the cmake interface. But isn't it confusing that a "set" attribute is named *Flags when we have many other *Flags attributes that cannot reasonably be sets? I'm thinking about configureFlags, qmakeFlags, makeFlags, sconsFlags etc. Maybe call the "set" variant I really like the other parts of this PR though, so I'd hate to see that list -> set conversion slow down a merge. |
They can already be either strings or lists depending on the *Flags. I don't see why adding another kind of *Flags increases confusion. If you grep for usage examples of I can live with another name, however |
|
I think continuing to use In case we do decide to use another name, I think |
I'm not in favor of the implementation ( |
I would not want to couple Nix to a specific builder (i.e. bash). Is there a good way to do a performance benchmark to see the impact of this change before thinking about adding a new feature to Nix? |
I don't think this could be noticeable for performance of standard nix/nixpkgs/nixos usage. |
This is indeed a common idiom, but I do not really see a value in adding a new abstraction for it. Also, it does not cover all cases of overrideDerivation, as it cannot be used to remove an attribute, only replacing attributes with some dummy values. |
Also, overrideAttrs looks entirely unrealted to cmake, and I do not think this can be a blocker, and as such I will recommend making another pull request dedicated to this other patch, only. |
@nbp, can you explain how overrideDerivation can be used to remove an attribute? Also, since AFAIK The reason I have put it with the (oldAttrs: rec {
cmakeFlags = oldAttrs.cmakeFlags // {
some_cmake_flag = true;
};
})
Please take a closer look at the changes I made, including the commit messages, and let me know if something is still unclear. I am happy to put |
Maybe you can split the overrideAttrs to separate PR? You can then add some documentation to Nixpkgs manual, it already references override and overrideDerivation. Also there will be clash if/when #18455 is merged. |
I've iterated on the design for this locally and made an RFC: NixOS/rfcs#13. |
There are three main parts to this change: - Support a new `setupHookFunc` in `mkDerivation` - Using attribute sets for `cmakeFlags` in Nix - Passing `cmakeFlags` to the builder more smartly == Introducing `setupHookFunc` The `setupHook` attribute allows a derivation to register arbitrary modifications to run in the context of downstream dependents' builds. To enable similar modifications to be made to the attributes passed to `mkDerivation`, teach `mkDerivation` a new trick: `setupHookFunc`. A derivation can set `setupHookFunc` to a Nix function which will be called for downstream dependencies (TODO using the same logic as the setupHook inclusion), and which takes the attribute set which was passed to `mkDerivation` and returns a new attribute set with any desired changes. These will stack (TODO: resolution order?) for composability, and having these live alongside the original derivation avoids cluttering `mkDerivation` with cmake-specific code (which was done in an earlier iteration). TODO: benchmark == Using Attribute Sets for `cmakeFlags` Almost all flags passed to `cmakeFlags` are setting CMake cache options with "-D<name>=<value>" or "-D<name>:<type>=<value>". Because the CMake cache is essentially a set of keys and values, model `cmakeFlags` as an attribute set in Nix as well. The cmake `setupHookFunc` will take a `cmakeFlags` set and convert each key/value pair to an appropriate "-D" option for cmake. String values are used as is, while integers are converted to strings and booleans are converted to `"ON"` and `"OFF"`. This presents the following benefits: - Removal of "-D" visual noise everywhere for ease of reading. - The `=` operater is literal, allowing extra whitespace for more ease of reading. - CMake options are now first class Nix values that can be operated on directly with the full strength of the Nix language. For example, boolean values can be used directly because the serialization logic is consolidated into the cmake `setupHookFunc`, getting rid of helpers like `edf` and making it easier to re-use feature flags like `pythonSupport`. - Sets automatically sort options, improving cache re-use. - An override which updates a CMake flag option will only incur (amortized) O(1) cost with a set, as opposed to O(n) cost with an (unsorted) list, which would require a full linear search. It's also shorter to write: just use `//`. A few other CMake flag types are supported: - Setting a value to null will cause its key to be unset via "-U<name>". - The `generator` key is mapped to the "-G<value>" option. - The `extraArgs` key is an escape hatch; any flags in this list of strings are not pre-processed, but concatenated to the generated list. Because `setupHookFunc`s are processed during `mkDerivation`, when overriding `cmakeFlags` `overrideAttrs` should be used instead of `overrideDerivation` so the cmake `setupHookFunc` has a chance to rerun. == Passing `cmakeFlags` to the builder more smartly Nix's regular conversion of lists to an environment variable simply concatenates the contents with spaces, which breaks if any of the flags themselves contain spaces. `cmakeFlagsArray` was added as a way to get around this limitation, but is unsatisfactory to use because items must be appended via bash in a preConfigure hook. Instead, pass the list to the builder in a smarter way: instead of relying on Nix's conversion to a string, perform our own conversion by escaping double quotes, double quoting each item, and passing the flags as a Bash array, which is hydrated via `eval`. This handles flags with spaces, newlines and other whitespace, as well as double quotes, faithfully. TODO: does it handle eg some_var="$out/blah" Make the list available during preConfigure as a bash array, so any dynamic modifications to the CMake flags can be done there. These changes make also `cmakeFlagsArray` redundant, so it has been removed and replaced in all cases with `cmakeFlags`.
4e55707
to
e62f88b
Compare
After seeing #44423 merged, I decided to revisit this. The main change this time is the introduction of |
@@ -236,6 +243,11 @@ rec { | |||
]; | |||
__propagatedImpureHostDeps = computedPropagatedImpureHostDeps ++ __propagatedImpureHostDeps; | |||
}; | |||
# TODO: use dependencies/propagatedDependencies instead of just nativeBuildInputs? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Ericson2314, any help you could provide here as to the correct incantation/set of dependencies to use here would be appreciated. I used nativeBuildInputs
to prototype this as cmake
is always in nativeBuildInputs
, at least AFAIK.
Note to self for next update:
|
Yeah I've talked to @matthewbauer about how master...matthewbauer:meson-cross begs for this :). If you go full structured attrs, then I suppose this has featured-creaped into stdenv 2.0! I do want to land something, but I also do believe we need such a huge overhaul of stdenv, so I'm fine either way. CC @LnL7 |
I like the idea of using an attribute set, but I agree with the sentiment that this shouldn't happen during evaluation and probably doesn't belong in the stdenv (ie. structured attrs / stdenv2). |
Oh but I think absolutely as much as possible should happen during evaluation as possible. |
I'm talking about implementation logic like this, not the values themselves. If we replace builders with functions and dynamic strings we wouldn't need structure attrs at all. nixpkgs/pkgs/development/tools/build-managers/cmake/default.nix Lines 73 to 94 in e62f88b
|
Are there any updates on this pull request, please? |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/structured-cmakeflags-attrs/6261/1 |
Hello, I'm a bot and I thank you in the name of the community for your contributions. Nixpkgs is a busy repository, and unfortunately sometimes PRs get left behind for too long. Nevertheless, we'd like to help committers reach the PRs that are still important. This PR has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human. If this is still important to you and you'd like to remove the stale label, we ask that you leave a comment. Your comment can be as simple as "still important to me". But there's a bit more you can do: If you received an approval by an unprivileged maintainer and you are just waiting for a merge, you can @ mention someone with merge permissions and ask them to help. You might be able to find someone relevant by using Git blame on the relevant files, or via GitHub's web interface. You can see if someone's a member of the nixpkgs-committers team, by hovering with the mouse over their username on the web interface, or by searching them directly on the list. If your PR wasn't reviewed at all, it might help to find someone who's perhaps a user of the package or module you are changing, or alternatively, ask once more for a review by the maintainer of the package/module this is about. If you don't know any, you can use Git blame on the relevant files, or GitHub's web interface to find someone who touched the relevant files in the past. If your PR has had reviews and nevertheless got stale, make sure you've responded to all of the reviewer's requests / questions. Usually when PR authors show responsibility and dedication, reviewers (privileged or not) show dedication as well. If you've pushed a change, it's possible the reviewer wasn't notified about your push via email, so you can always officially request them for a review, or just @ mention them and say you've addressed their comments. Lastly, you can always ask for help at our Discourse Forum, or more specifically, at this thread or at #nixos' IRC channel. |
Further progress on this would require #72074. |
Motivation for this change
Similar to #15799, but with a smaller scope (can change everything at one time so no need for deprecation) and with an even nicer interface.
Things done
(nix.useChroot on NixOS,
or option
build-use-chroot
innix.conf
on non-NixOS)
nix-shell -p nox --run "nox-review wip"
./result/bin/
)There are three main parts to this change:
setupHookFunc
inmkDerivation
cmakeFlags
in NixcmakeFlags
to the builder more smartly== Introducing
setupHookFunc
The
setupHook
attribute allows a derivation to register arbitrarymodifications to run in the context of downstream dependents' builds.
To enable similar modifications to be made to the attributes passed
to
mkDerivation
, teachmkDerivation
a new trick:setupHookFunc
.A derivation can set
setupHookFunc
to a Nix functionwhich will be called for downstream dependencies
(TODO using the same logic as the setupHook inclusion),
and which takes the attribute set which was passed to
mkDerivation
and returns a new attribute set with any desired changes.
These will stack (TODO: resolution order?) for composability,
and having these live alongside the original derivation
avoids cluttering
mkDerivation
with cmake-specific code(which was done in an earlier iteration).
TODO: benchmark
== Using Attribute Sets for
cmakeFlags
Almost all flags passed to
cmakeFlags
are setting CMake cache optionswith "-D=" or "-D:=".
Because the CMake cache is essentially a set of keys and values,
model
cmakeFlags
as an attribute set in Nix as well.The cmake
setupHookFunc
will take acmakeFlags
setand convert each key/value pair to an appropriate "-D" option for cmake.
String values are used as is, while integers are converted to strings
and booleans are converted to
"ON"
and"OFF"
.This presents the following benefits:
=
operater is literal, allowing extra whitespace for more easeof reading.
directly with the full strength of the Nix language.
For example, boolean values can be used directly because the
serialization logic is consolidated into the cmake
setupHookFunc
,getting rid of helpers like
edf
and making it easier to re-use feature flags like
pythonSupport
.(amortized) O(1) cost with a set, as opposed to O(n) cost with an
(unsorted) list, which would require a full linear search.
It's also shorter to write: just use
//
.A few other CMake flag types are supported:
generator
key is mapped to the "-G" option.extraArgs
key is an escape hatch; any flags in this list ofstrings are not pre-processed, but concatenated to the generated list.
Because
setupHookFunc
s are processed duringmkDerivation
,when overriding
cmakeFlags
overrideAttrs
should be used instead ofoverrideDerivation
so the cmakesetupHookFunc
has a chance to rerun.== Passing
cmakeFlags
to the builder more smartlyNix's regular conversion of lists to an environment variable
simply concatenates the contents with spaces,
which breaks if any of the flags themselves contain spaces.
cmakeFlagsArray
was added as a way to get around this limitation,but is unsatisfactory to use because items must be appended via bash
in a preConfigure hook.
Instead, pass the list to the builder in a smarter way:
instead of relying on Nix's conversion to a string,
perform our own conversion by escaping double quotes,
double quoting each item, and passing the flags as a Bash array,
which is hydrated via
eval
.This handles flags with spaces, newlines and other whitespace,
as well as double quotes, faithfully.
TODO: does it handle eg some_var="$out/blah"
Make the list available during preConfigure as a bash array, so any
dynamic modifications to the CMake flags can be done there.
These changes make also
cmakeFlagsArray
redundant, so it has beenremoved and replaced in all cases with
cmakeFlags
.Notes/questions: