-
-
Notifications
You must be signed in to change notification settings - Fork 638
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Macros should be able to consume the engine to request files/processes #7022
Comments
I've updated the title of this one, because it applies to
|
I believe I've just hit this:
Running Full output below.
|
@TansyArron : Were you using EDIT: Ah, yes: according to the stack, you are. Cool! So, the workaround for this one is something like:
|
The invalidation globs for pantsd (improved in #5567) can be manually configured to cover everything that is known to cause pantsd to need to restart. But many of the things that should trigger a restart are already known to pants. A partial list: 1. The content of any configured pants.ini files (due to #7022) 2. The pythonpath of pants itself (since changes to loose-source plugins mean the code that pantsd is running might have changed) We should automatically include these values (and likely others!) in the values that we use for --invalidation-globs. In cases where the options values point to files that exist outside of the buildroot, we should consider logging a warning (unless that ends up being too noisy). Fixes #7595.
…ld#7022. # Delete this line to force CI to run Clippy and the Rust tests. [ci skip-rust-tests] # Delete this line to force CI to run the JVM tests. [ci skip-jvm-tests]
…ld#7022. [ci skip-rust-tests] [ci skip-jvm-tests]
This continues to affect pantsd, but I'm going to get a PR out that bakes in the workaround mentioned in #7022 (comment) to avoid it being a launch blocker. |
…ld#7022. [ci skip-rust-tests] [ci skip-jvm-tests]
…9946) ### Problem As explained in #7022, macros that open files can have untracked file dependencies that `pantsd` cannot account for. There is only one (relatively common) case of this: the `python_requirements` macro (which creates a target per line in a `requirements.txt` file). ### Solution We need to address #7022 in a fundamental way before 2.0. But for now we can ameliorate the only known case by defaulting to invalidating for `requirements.txt` files. [ci skip-rust-tests] [ci skip-jvm-tests]
…9946) ### Problem As explained in #7022, macros that open files can have untracked file dependencies that `pantsd` cannot account for. There is only one (relatively common) case of this: the `python_requirements` macro (which creates a target per line in a `requirements.txt` file). ### Solution We need to address #7022 in a fundamental way before 2.0. But for now we can ameliorate the only known case by defaulting to invalidating for `requirements.txt` files. [ci skip-rust-tests] [ci skip-jvm-tests]
Moving discussion from #11724 (cc @jsirois, @Eric-Arellano):
Yep. Mentioned this on the linked ticket: #10723. Macros implemented as As it stands, finishing this ticket would allow for fixing the |
That's great for extending Pants more easily? But I'm stuck on a different issue. Rules consume field sets, field sets currently come from targets parsed from BUILD files by rules. There is no reason I can see we can't parse more file types to produce field sets for rules to consume. Notice there is no mention of or need for macros here. |
EDIT: So... yes, I think that we're on the same page. The challenge is seemingly just knowing "which noun/identifier you hang the |
To continue the thought from above... one way to hook in dynamic "creation" of targets/ |
OK. What I hear from all that is yes, we can parse files and produce field sets for rules to consume (since we already do). There is no engine change, there are only rule changes. Our current rules are too restrictive in this sense and so that may make the change harder than it otherwise might be - but its fundamentally doable. |
Another usecase arose recently: that of invoking |
I'm helping a friend set up Pants in their monorepo(!), and to repeat what's known: this is a real pain point. It tripped me up for a solid 2 minutes why Pants wasn't seeing a change to |
@tdyas and I had a strawman design session on this today in the context of the Go plugin and agreed for the need for proper "target generation" as a first class citizen of the engine. Encapsulation:
Timing of generation:
Capabilities needed for generation rules:
-- We also discussed that this change could unlock some amazing benefits for Pants, especially boilerplate reduction! We could essentially implement
But worth pointing out that that may still be valuable. Rather than leaning into target gen, perhaps a redesign of targets is more important. |
To be more precise, I meant that some synthetic targets may be usable as input to target generation rules, not that a synthetic target necessarily itself generates another synthetic target. Using the Go plugin as an example, target generation might happen as follows:
There is no explicit ordering between those rules right now. A basic implemention could run an iterative algorithm to keep applying target generation rules until no more targets are added. |
Ah got it. To check my understanding: if the user explicitly specified a I think that's possible w/o needing to add special ordering. You'd have two rules for each target gen, and the @rule
def generate_go_ext_mod_packages(request: GenGoExtModPackages) -> SyntheticTargets:
# invoke go code
return SyntheticTargets([_GoExtModPackage(...), ...])
@rule
def generate_go_external_module(request: GenGoExternalModule) -> SyntheticTargets:
external_module_tgt = GoExternalModule(...)
mod_packages = await Get(SyntheticTargets, GenGoExtModPackages(external_module))
return SyntheticTargets([external_module_tgt, *mod_packages]) The generic "target gen" code will invoke both rules based on what it detects from what's on disk, as both will be registered with unions. No need for special ordering, and it's very explicit in each rule what will be generated. The key insight is having a return type of |
Correct. If the user wants to supply a |
) Go modules consist of multiple packages (which are compiled one package at a time). In order to support building third-party module source code, the Go plugin needs to understand the dependencies between packages in those modules and dependencies on packages not in those sources. First-party Go sources already have an associated `go_package` target type that can have dependencies on other first-party code. This PR adds a `_go_ext_mod_package` target type to similarly represent packages in third-party sources (and allow those external packages to be dependencies in the build graph of both first-party and third-party Go sources). This new target type is generated by `./pants tailor` for now, but will become an implementation detail not visible to users once synthetic target generation is implemented. (See #7022 for full discussion of synthetic target generation ideas.) This PR does not add dependency inference on packages in third-party modules. That will be in the immediate follow-up PR.
@Eric-Arellano added a new implementation for macros based around target generation! Am going to close this one in favor of #12915, which discusses using the new macros to fix the problem with |
…acro in favor of target generation (#14075) Closes #12915. ## Benefits of switching to target generation * Closes #7022. * Target generators show up in `./pants help`. * Consistent mechanism for target generation, as explained at https://www.pantsbuild.org/docs/targets. ## Deprecation plan This PR adds a new global option `--use-deprecated-python-macros` with a default of `True`. When True, `parser.py` is taught to load the old macros; else it loads the new targets. The PR deprecates the default of `--use-deprecated-python-macros` so that it will default to `False` in Pants 2.11. Then, we'll deprecate the option outright and remove it in Pants 2.12, meaning the macros will be removed then too. ## How users upgrade ``` ❯ ./pants 17:06:21.30 [WARN] DEPRECATED: the option `--use-deprecated-python-macros` defaulting to true will be removed in version 2.11.0.dev0. In Pants 2.11, the default for the global option `--use-deprecated-python-macros` will change to false. To fix this deprecation, explicitly set `use_deprecated_python_macros = true` in the `[GLOBAL]` section of `pants.toml`. Or, When you are ready to upgrade to the improved target generation mechanism, follow these steps: 1. Run `./pants update-build-files --fix-python-macros` 2. Check the logs for an ERROR log to see if you have to manually add `name=` anywhere. 3. Set `use_deprecated_python_macros = false` in `[GLOBAL]` in pants.toml. (Why upgrade from the old macro mechanism to target generation? Among other benefits, it makes sure that the Pants daemon is properly invalidated when you change `requirements.txt` and `pyproject.toml`.) ```
Multiple problems stem from the v1 macro system (
context_aware_object_factory
) not being able to access the engine. We should likely replace it with an engine-aware macro system, and/or generally add support in the engine APIs for generating multiple targets from one target/macro.python_requirements is a
context_aware_object_factory
that declares multiple targets by reading arequirements.txt
file adjacent to its BUILD file.While its invalidation was recently (#6405) improved, the fact that execution of that macro means that BUILD file parsing depends on the
requirements.txt
file is still not declared (rather, we declare that any targets that the macro defines depend on therequirements.txt
file, which is also true). This means that adding a new requirement to therequirements.txt
file does not trigger re-parsing of the BUILD file to discover the new target that should be created.Fixing this bug relates to a host of other issues:
python_requirements()
does.In Specifying deps outside of BUILD files (aka "dep inference") #6449, we discussed giving BUILD file parsing the ability to consume other nearby files to accomplish its work.In #6998, @illicitonion mentioned accessing
@rules
via BUILD file parsing. While this is not currently possible, it definitely suggests a possible solution to both this bug and to #3561: namely, that we could design an API to replace macros/CAOFs/objects with@rules
that produce one or more BUILD symbols. Meanwhile, the use of@rules
for this case might also inspire a solution to #6449.The text was updated successfully, but these errors were encountered: