-
Notifications
You must be signed in to change notification settings - Fork 29.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
src: refactor options parsing #22392
Conversation
This is a major refactor of our Node’s parser. See `node_options.cc` for how it is used, and `node_options-inl.h` for the bulk of its implementation. Unfortunately, the implementation has come to have some complexity, in order to meet the following goals: - Make it easy to *use* for defining or changing options. - Keep it (mostly) backwards-compatible. - No tests were harmed as part of this commit. - Be as consistent as possible. - In particular, options can now generally accept arguments through both `--foo=bar` notation and `--foo bar` notation. We were previously very inconsistent on this point. - Separate into different levels of scope, namely per-process (global), per-Isolate and per-Environment (+ debug options). - Allow programmatic accessibility in the future. - This includes a possible expansion for `--help` output. This commit also leaves a number of `TODO` comments, mostly for improving consistency even more (possibly with having to modify tests), improving embedder support, as well as removing pieces of exposed configuration variables that should never have become part of the public API but unfortunately are at this point.
@@ -72,35 +72,38 @@ static void Initialize(Local<Object> target, | |||
READONLY_BOOLEAN_PROPERTY("hasTracing"); | |||
#endif | |||
|
|||
READONLY_STRING_PROPERTY(target, "icuDataDir", icu_data_dir); | |||
// TODO(addaleax): This seems to be an unused, private API. Remove it? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I remember @srl295 saying this might be used by embedders.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It’s only accessible through process.binding('config')
, so it’s still going to have to change somehow, assuming we push through with the process.binding()
deprecations…
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I love it! Way way way overdue refactoring.
But I don't like that OptionsParser
is dynamically extensible. AFAIK it's a well established wheel, so either we assume an existing implementation, or make it static.
// These methods add a single option to the parser. Optionally, it can be | ||
// specified whether the option should be allowed from environment variable | ||
// sources (i.e. NODE_OPTIONS). | ||
void AddOption(const std::string& name, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why can't these be templates?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The way this is done is because we use it to infer the info.type
value. We could do something with templates but I think it would mean we’d still use a list of template specializations?
I don’t really understand this comment, can you explain or give examples? |
src/node_options.cc
Outdated
|
||
namespace node { | ||
|
||
DebugOptionsParser::DebugOptionsParser() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAIR this could be merged with EnvironmentOptionsParser
. It was encapsulated as in preparation to something like this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this could be merged. As you are saying, it helps with encapsulation, so I don’t mind it either way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would add a comment stating this, just so that future generations won't assume it's something special.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good, done! (3c9b03a)
Fantastic start on this. One the things I would really like to see is a quick way of determining if the options have changed from one run to another (essentially a hash value calculated from the options values) |
@jasnell Just so I understand it, something that one could get today |
Yes and no. Env vars also need to be taken into consideration and input should be normalized so that order doesn't matter (except in those handful of cases where it does, of course). It also needs to take into consideration default values that may happen to change. |
I mean implementing a generic agrv parser. I don't have enough production experience with any specific library but https://github.com/search?l=C%2B%2B&q=argument+parser&type=Repositories show afew with > 100 stars. AddOption("--inspect-port", &DebugOptions::host_port,
kAllowedInEnvironment);
AddAlias("--debug-port", "--inspect-port");
AddOption("--inspect", &DebugOptions::inspector_enabled,
kAllowedInEnvironment);
AddAlias("--inspect=", { "--inspect-port", "--inspect" });
AddOption("--debug", &DebugOptions::deprecated_debug);
AddAlias("--debug=", { "--inspect-port", "--debug" }); we would have something like Option<"inspect-port", kAllowedInEnvironment> host_port;
Alias<"inspect-port"> debug-port; that might requires reflection, or c'tors with side effects, or macros... So in that sense your implementation is quite minimal, it's just that most of that information is known in compile time, so there no need to delegate it to run time. |
I think that’s not doable given how weird some of our options are (think debug flags or
I agree that there is unnecessary runtime overhead here that I’d like to get down a bit, yes. 👍 |
bool Options::* field, | ||
OptionEnvvarSettings env_setting) { | ||
options_.emplace(name, OptionInfo { | ||
kBoolean, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just thinking out loud... (It is an interesting problem after all).
How about instead on a const
you put a pointer to the conversion function. That way you make these Templates specializations on the typeof(field)
, and eliminate the switch in L385.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Had a few more "mind itches". I'll try to write up a draft.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Had a few more "mind itches". I'll try to write up a draft.
Sounds good, looking forward to it!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Can you also do a CITGM run?
Yup, sounds like a good idea: CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1517/ |
@boneskull If this PR stays in its current shape, it shouldn’t be too hard – we could go through the |
@refack Ping … anything you’d definitely like to see changed before merging this? It would be nice to get this in and work on expanding it (programmatic accessibility/help texts). :) |
No reason to wait. If I find the time we could iterate later. |
|
@refack I think it might just be that New CI: https://ci.nodejs.org/job/node-test-pull-request/16655/ |
|
||
inline int ParseAndValidatePort(const std::string& port, std::string* error) { | ||
char* endptr; | ||
errno = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't this lint? How did this compile before? Ahh the wonder of C++ compilers...
And in the next line could we switch to strtoul
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lint? It’s just a missing include, I assume… and, yes, I think we could switch, but it shouldn’t make any difference?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It’s just a missing include
ahh it's defined in errno.h
(that's a POSIX mechanism). Without that context it does look like an undefined var (who's value magicly changes).
This is a major refactor of our Node’s parser. See `node_options.cc` for how it is used, and `node_options-inl.h` for the bulk of its implementation. Unfortunately, the implementation has come to have some complexity, in order to meet the following goals: - Make it easy to *use* for defining or changing options. - Keep it (mostly) backwards-compatible. - No tests were harmed as part of this commit. - Be as consistent as possible. - In particular, options can now generally accept arguments through both `--foo=bar` notation and `--foo bar` notation. We were previously very inconsistent on this point. - Separate into different levels of scope, namely per-process (global), per-Isolate and per-Environment (+ debug options). - Allow programmatic accessibility in the future. - This includes a possible expansion for `--help` output. This commit also leaves a number of `TODO` comments, mostly for improving consistency even more (possibly with having to modify tests), improving embedder support, as well as removing pieces of exposed configuration variables that should never have become part of the public API but unfortunately are at this point. PR-URL: #22392 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gus Caplan <[email protected]>
I, uhm, might have messed up by using a `substr(start, end)` signature when `std::string` actually uses `substr(start, len)`. Fix that. Fixes: nodejs#22526 Refs: nodejs#22392
I, uhm, might have messed up by using a `substr(start, end)` signature when `std::string` actually uses `substr(start, len)`. Fix that. Fixes: #22526 Refs: #22392 PR-URL: #22529 Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]>
This is a major refactor of our Node’s parser. See `node_options.cc` for how it is used, and `node_options-inl.h` for the bulk of its implementation. Unfortunately, the implementation has come to have some complexity, in order to meet the following goals: - Make it easy to *use* for defining or changing options. - Keep it (mostly) backwards-compatible. - No tests were harmed as part of this commit. - Be as consistent as possible. - In particular, options can now generally accept arguments through both `--foo=bar` notation and `--foo bar` notation. We were previously very inconsistent on this point. - Separate into different levels of scope, namely per-process (global), per-Isolate and per-Environment (+ debug options). - Allow programmatic accessibility in the future. - This includes a possible expansion for `--help` output. This commit also leaves a number of `TODO` comments, mostly for improving consistency even more (possibly with having to modify tests), improving embedder support, as well as removing pieces of exposed configuration variables that should never have become part of the public API but unfortunately are at this point. PR-URL: nodejs#22392 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Gus Caplan <[email protected]>
This is a major refactor of our Node’s parser. See `node_options.cc` for how it is used, and `node_options-inl.h` for the bulk of its implementation. Unfortunately, the implementation has come to have some complexity, in order to meet the following goals: - Make it easy to *use* for defining or changing options. - Keep it (mostly) backwards-compatible. - No tests were harmed as part of this commit. - Be as consistent as possible. - In particular, options can now generally accept arguments through both `--foo=bar` notation and `--foo bar` notation. We were previously very inconsistent on this point. - Separate into different levels of scope, namely per-process (global), per-Isolate and per-Environment (+ debug options). - Allow programmatic accessibility in the future. - This includes a possible expansion for `--help` output. This commit also leaves a number of `TODO` comments, mostly for improving consistency even more (possibly with having to modify tests), improving embedder support, as well as removing pieces of exposed configuration variables that should never have become part of the public API but unfortunately are at this point. PR-URL: #22392 Backport-PR-URL: #22558 Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>
I, uhm, might have messed up by using a `substr(start, end)` signature when `std::string` actually uses `substr(start, len)`. Fix that. Fixes: #22526 Refs: #22392 PR-URL: #22529 Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]>
This is a major refactor of our Node’s parser. See `node_options.cc` for how it is used, and `node_options-inl.h` for the bulk of its implementation. Unfortunately, the implementation has come to have some complexity, in order to meet the following goals: - Make it easy to *use* for defining or changing options. - Keep it (mostly) backwards-compatible. - No tests were harmed as part of this commit. - Be as consistent as possible. - In particular, options can now generally accept arguments through both `--foo=bar` notation and `--foo bar` notation. We were previously very inconsistent on this point. - Separate into different levels of scope, namely per-process (global), per-Isolate and per-Environment (+ debug options). - Allow programmatic accessibility in the future. - This includes a possible expansion for `--help` output. This commit also leaves a number of `TODO` comments, mostly for improving consistency even more (possibly with having to modify tests), improving embedder support, as well as removing pieces of exposed configuration variables that should never have become part of the public API but unfortunately are at this point. PR-URL: #22392 Backport-PR-URL: #22558 Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>
I, uhm, might have messed up by using a `substr(start, end)` signature when `std::string` actually uses `substr(start, len)`. Fix that. Fixes: #22526 Refs: #22392 PR-URL: #22529 Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]>
Notable changes: * child_process: * `TypedArray` and `DataView` values are now accepted as input by `execFileSync` and `spawnSync`. #22409 * coverage: * Native V8 code coverage information can now be output to disk by setting the environment variable `NODE_V8_COVERAGE` to a directory. #22527 * deps: * The bundled npm was upgraded to version 6.4.1. #22591 * Changelogs: [6.3.0-next.0](https://github.com/npm/cli/releases/tag/v6.3.0-next.0) [6.3.0](https://github.com/npm/cli/releases/tag/v6.3.0) [6.4.0](https://github.com/npm/cli/releases/tag/v6.4.0) [6.4.1](https://github.com/npm/cli/releases/tag/v6.4.1) * fs: * The methods `fs.read`, `fs.readSync`, `fs.write`, `fs.writeSync`, `fs.writeFile` and `fs.writeFileSync` now all accept `TypedArray` and `DataView` objects. #22150 * A new boolean option, `withFileTypes`, can be passed to to `fs.readdir` and `fs.readdirSync`. If set to true, the methods return an array of directory entries. These are objects that can be used to determine the type of each entry and filter them based on that without calling `fs.stat`. #22020 * http2: * The `http2` module is no longer experimental. #22466 * os: * Added two new methods: `os.getPriority` and `os.setPriority`, allowing to manipulate the scheduling priority of processes. #22394 * process: * Added `process.allowedNodeEnvironmentFlags`. This object can be used to programmatically validate and list flags that are allowed in the `NODE_OPTIONS` environment variable. #19335 * src: * Deprecated option variables in public C++ API. #22392 * Refactored options parsing. #22392 * vm: * Added `vm.compileFunction`, a method to create new JavaScript functions from a source body, with options similar to those of the other `vm` methods. #21571 * Added new collaborators: * [lundibundi](https://github.com/lundibundi) - Denys Otrishko PR-URL: #22716
This is a major refactor of our Node’s parser. See `node_options.cc` for how it is used, and `node_options-inl.h` for the bulk of its implementation. Unfortunately, the implementation has come to have some complexity, in order to meet the following goals: - Make it easy to *use* for defining or changing options. - Keep it (mostly) backwards-compatible. - No tests were harmed as part of this commit. - Be as consistent as possible. - In particular, options can now generally accept arguments through both `--foo=bar` notation and `--foo bar` notation. We were previously very inconsistent on this point. - Separate into different levels of scope, namely per-process (global), per-Isolate and per-Environment (+ debug options). - Allow programmatic accessibility in the future. - This includes a possible expansion for `--help` output. This commit also leaves a number of `TODO` comments, mostly for improving consistency even more (possibly with having to modify tests), improving embedder support, as well as removing pieces of exposed configuration variables that should never have become part of the public API but unfortunately are at this point. PR-URL: #22392 Backport-PR-URL: #22558 Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Daniel Bevenius <[email protected]>
I, uhm, might have messed up by using a `substr(start, end)` signature when `std::string` actually uses `substr(start, len)`. Fix that. Fixes: #22526 Refs: #22392 PR-URL: #22529 Reviewed-By: Gus Caplan <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]>
Notable changes: * child_process: * `TypedArray` and `DataView` values are now accepted as input by `execFileSync` and `spawnSync`. #22409 * coverage: * Native V8 code coverage information can now be output to disk by setting the environment variable `NODE_V8_COVERAGE` to a directory. #22527 * deps: * The bundled npm was upgraded to version 6.4.1. #22591 * Changelogs: [6.3.0-next.0](https://github.com/npm/cli/releases/tag/v6.3.0-next.0) [6.3.0](https://github.com/npm/cli/releases/tag/v6.3.0) [6.4.0](https://github.com/npm/cli/releases/tag/v6.4.0) [6.4.1](https://github.com/npm/cli/releases/tag/v6.4.1) * fs: * The methods `fs.read`, `fs.readSync`, `fs.write`, `fs.writeSync`, `fs.writeFile` and `fs.writeFileSync` now all accept `TypedArray` and `DataView` objects. #22150 * A new boolean option, `withFileTypes`, can be passed to to `fs.readdir` and `fs.readdirSync`. If set to true, the methods return an array of directory entries. These are objects that can be used to determine the type of each entry and filter them based on that without calling `fs.stat`. #22020 * http2: * The `http2` module is no longer experimental. #22466 * os: * Added two new methods: `os.getPriority` and `os.setPriority`, allowing to manipulate the scheduling priority of processes. #22407 * process: * Added `process.allowedNodeEnvironmentFlags`. This object can be used to programmatically validate and list flags that are allowed in the `NODE_OPTIONS` environment variable. #19335 * src: * Deprecated option variables in public C++ API. #22515 * Refactored options parsing. #22392 * vm: * Added `vm.compileFunction`, a method to create new JavaScript functions from a source body, with options similar to those of the other `vm` methods. #21571 * Added new collaborators: * [lundibundi](https://github.com/lundibundi) - Denys Otrishko PR-URL: #22716
Notable changes: * child_process: * `TypedArray` and `DataView` values are now accepted as input by `execFileSync` and `spawnSync`. #22409 * coverage: * Native V8 code coverage information can now be output to disk by setting the environment variable `NODE_V8_COVERAGE` to a directory. #22527 * deps: * The bundled npm was upgraded to version 6.4.1. #22591 * Changelogs: [6.3.0-next.0](https://github.com/npm/cli/releases/tag/v6.3.0-next.0) [6.3.0](https://github.com/npm/cli/releases/tag/v6.3.0) [6.4.0](https://github.com/npm/cli/releases/tag/v6.4.0) [6.4.1](https://github.com/npm/cli/releases/tag/v6.4.1) * fs: * The methods `fs.read`, `fs.readSync`, `fs.write`, `fs.writeSync`, `fs.writeFile` and `fs.writeFileSync` now all accept `TypedArray` and `DataView` objects. #22150 * A new boolean option, `withFileTypes`, can be passed to to `fs.readdir` and `fs.readdirSync`. If set to true, the methods return an array of directory entries. These are objects that can be used to determine the type of each entry and filter them based on that without calling `fs.stat`. #22020 * http2: * The `http2` module is no longer experimental. #22466 * os: * Added two new methods: `os.getPriority` and `os.setPriority`, allowing to manipulate the scheduling priority of processes. #22407 * process: * Added `process.allowedNodeEnvironmentFlags`. This object can be used to programmatically validate and list flags that are allowed in the `NODE_OPTIONS` environment variable. #19335 * src: * Deprecated option variables in public C++ API. #22515 * Refactored options parsing. #22392 * vm: * Added `vm.compileFunction`, a method to create new JavaScript functions from a source body, with options similar to those of the other `vm` methods. #21571 * Added new collaborators: * [lundibundi](https://github.com/lundibundi) - Denys Otrishko PR-URL: #22716
Notable changes: * child_process: * `TypedArray` and `DataView` values are now accepted as input by `execFileSync` and `spawnSync`. #22409 * coverage: * Native V8 code coverage information can now be output to disk by setting the environment variable `NODE_V8_COVERAGE` to a directory. #22527 * deps: * The bundled npm was upgraded to version 6.4.1. #22591 * Changelogs: [6.3.0-next.0](https://github.com/npm/cli/releases/tag/v6.3.0-next.0) [6.3.0](https://github.com/npm/cli/releases/tag/v6.3.0) [6.4.0](https://github.com/npm/cli/releases/tag/v6.4.0) [6.4.1](https://github.com/npm/cli/releases/tag/v6.4.1) * fs: * The methods `fs.read`, `fs.readSync`, `fs.write`, `fs.writeSync`, `fs.writeFile` and `fs.writeFileSync` now all accept `TypedArray` and `DataView` objects. #22150 * A new boolean option, `withFileTypes`, can be passed to to `fs.readdir` and `fs.readdirSync`. If set to true, the methods return an array of directory entries. These are objects that can be used to determine the type of each entry and filter them based on that without calling `fs.stat`. #22020 * http2: * The `http2` module is no longer experimental. #22466 * os: * Added two new methods: `os.getPriority` and `os.setPriority`, allowing to manipulate the scheduling priority of processes. #22407 * process: * Added `process.allowedNodeEnvironmentFlags`. This object can be used to programmatically validate and list flags that are allowed in the `NODE_OPTIONS` environment variable. #19335 * src: * Deprecated option variables in public C++ API. #22515 * Refactored options parsing. #22392 * vm: * Added `vm.compileFunction`, a method to create new JavaScript functions from a source body, with options similar to those of the other `vm` methods. #21571 * Added new collaborators: * [lundibundi](https://github.com/lundibundi) - Denys Otrishko PR-URL: #22716
In case anybody was wondering what I’ve been up to this week. 😸
This is a major refactor of our Node’s parser. See
node_options.cc
for how it is used, and
node_options-inl.h
for the bulkof its implementation.
Unfortunately, the implementation has come to have some
complexity, in order to meet the following goals:
through both
--foo=bar
notation and--foo bar
notation.We were previously very inconsistent on this point.
(Labelling semver-minor because of this).
per-process (global), per-Isolate and per-Environment
(+ debug options).
--help
output.This commit also leaves a number of
TODO
comments, mostly forimproving consistency even more (possibly with having to modify
tests), improving embedder support, as well as removing pieces of
exposed configuration variables that should never have become
part of the public API but unfortunately are at this point.
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passesFyi @boneskull, you probably want to be aware of this because of #19335. :)