Add API for root requests with multiple parameters

[stuhood]

In order to implement #5869, root requests will need to provide multiple input Params. #6170 added the ability for Nodes to have multiple parameters, but did not yet expose that via Get or Scheduler.product_request.

This issue should add the ability to make a root product request with multiple Params, but it should probably not expose that ability in the context of yield Get (since that does not block #5869, afaik).

To avoid a tight coupling where callers need to know exactly what Params their callees need (both in a root Scheduler.product_request call and in a Get), Params which are not used by a subgraph should be ignored. As an example, the OptionsParseRequest that will be passed into @console_rule execution might be ignored by a particular subgraph... but the caller should not need to know that, which will allow it to safely pass it for every Scheduler.product_request.

This issue should also handle updating the engine README for the new UX, and replacing the mentions of Variants.

---

This PR will primarily involve addressing a few TODOs from #6170:

• Safely constructing instances of Params containing keys with distinct TypeIds will require adding a constructor that validates that the input Keys' types are distinct (and failing politely otherwise).

• Methods between Scheduler.product_request and the TODO in RuleGraph::find_root_edges will need to be updated in order to pass down Params collections, rather than individual "subject" Keys.

• All usage of the Params::expect_single method should be replaced with consuming the relevant Param: since Params contain distinct Keys per type, this will usually involve calling Params::find(type_we_need) and asserting that it is present.

• Finally, to avoid propagating Params into subgraphs where they are not required, all of the constructors of Select should use rule_graph::Entry::params() to filter the set of Params (probably by adding a helper method to Params) that they are given to just what is actually required by the Entry. This will resolve a TODO in the Select constructors and half of the TODO in gen_get (the other half will be resolved when we add UX for passing multiple Params with Get).

---

At the end of this PR, you should be able to call something like:

# the subjects argument becomes a params argument which is a list of "params" dicts:
scheduler.product_request(HydratedTargets, [{Specs: Specs(..), OptionsParseRequest: OptionsParseRequest(..)}, ..])
# OR
# the subjects argument becomes a params argument which is a list of "params" tuples:
scheduler.product_request(HydratedTargets, [(Specs(..), OptionsParseRequest(..)), ..])
# OR
# the subjects argument becomes a params argument which is a list of a new Params datatype:
scheduler.product_request(HydratedTargets, [Params(Specs(..), OptionsParseRequest(..)), ..])
# etc

...to trigger a @rule that requires two inputs.

[stuhood]

Considered working on this for a bit, but given the time I put into expanding the description, it would be good if someone else could grab it this week.

[ity]

@stuhood trying to show this with an example. at the end of this change, we will go from:

from (subject, product) pairs:
subject = Specs(dependencies=(SingleAddress(directory=u'', name=u'compiler-interface'),),matcher=SpecsMatcher(tags=(), exclude_patterns=()))
product = <class 'pants.engine.legacy.graph.TransitiveHydratedTargets'>]

to (list of subject/params, product) pairs:
subject = [Specs(dependencies=(SingleAddress(directory=u'', name=u'compiler-interface'))), Specs(dependencies=(SingleAddress(directory=u'', name=u'zinc-extractor'))]
product = <class 'pants.engine.legacy.graph.TransitiveHydratedTargets'>]

So, on the Python side we would have to accumulate the subject entries in a way that they are distinct before they are even crossing over?

will require adding a constructor that validates that the input Keys' types are distinct (and failing politely otherwise).

or when writing this ^ were you thinking of letting the invocations come through and have validation later?

[stuhood]

at the end of this change, we will go from...

Hm... I don't think so.

It's more like going from our current API, which is:

# the subjects argument is a list of a subject values
scheduler.product_request(HydratedTargets, [Specs(..), ..])

to one of the syntaxes mentioned in the description. Expanding them a bit here and using HydratedTargets and Specs:

# the subjects argument becomes a params argument which is a list of "params" dicts:
scheduler.product_request(HydratedTargets, [{Specs: Specs(..), OptionsParseRequest: OptionsParseRequest(..)}, ..])
# OR
# the subjects argument becomes a params argument which is a list of "params" tuples:
scheduler.product_request(HydratedTargets, [(Specs(..), OptionsParseRequest(..)), ..])
# OR
# the subjects argument becomes a params argument which is a list of a new Params datatype:
scheduler.product_request(HydratedTargets, [Params(Specs(..), OptionsParseRequest(..)), ..])
# etc

So, on the Python side we would have to accumulate the subject entries in a way that they are distinct before they are even crossing over?

No: it's only the members of an individual "Params" instance that need to be distinct. In the expanded example above, one instance of Params contains Specs and OptionsParseRequest (which are distinct types).

or when writing this ^ were you thinking of letting the invocations come through and have validation later?

Since this can fail if I were to do something like Params(Specs(..), Specs(..)) (ie, one set of Params with two instances of the same type), we do need to do validation. But if you want to just have it panic for now, we can leave a TODO about cleaning it up.

[ity]

this got put on the backburner in lieu of other higher priority issues - picking this back up tomorrow

[stuhood]

Thank you!

[ity]

i will get this branch to an unfinished but working state this weekend and post an update to this so its easier for anyone who wants to pick it up

[ity]

Added my old wip branch here - https://github.com/ity/pants/tree/ity/p3 If I get some more time today, I will clean it up, otherwise these are all the changes I had so far against this issue.

[stuhood]

#6766 landed to address the third and fourth paragraph from the description. Have converted to checkboxes and ticked them to make that clear.

I'm going to put this down for the evening to more directly investigate what blocks demoability.

[stuhood]

I'm going to put this down for the evening to more directly investigate what blocks demoability.

Annnd, after a brief investigation, it looks like this might still be the largest blocker for demoability. Resuming.

[stuhood]

I might start tackling #5869 concurrently with this issue, in order to better experiment with the API.

[stuhood]

Putting this down, because it looks like #5869 is not blocked on it.