[Search Pipelines] Apply default search pipelines to requests that don't target a single index

[msfroh]

Is your feature request related to a problem? Please describe.
In #7470, I took a bit of a shortcut, using an index setting to define a default search pipeline, copying the idea from ingest pipelines, where the concept is easier since an IndexRequest targets a single index and a BulkRequest is just a batch of IndexRequests.

For search, the problem is often more equally complicated. On search, you have index patterns and index aliases. At indexing time, there is logic to resolve index aliases and templates --search doesn't worry about index templates, but it does need to worry about index patterns, so really, the problem is equally complicated for indexing and search.

Describe the solution you'd like
I think search patterns and search aliases could follow a similar pattern to resolution: expand the pattern/alias to the target index(es), and run the default search pipelines associated with each target index. We may need to send a variety of ShardSearchRequests that differ by more than shard alone, depending on how the pipeline modified the SearchRequest. A (possibly difficult) challenge is that this also means changing the reduce logic, at least on the fetch/expand phases, since they currently produce a single SearchResponse aggregated across all shards from all indices matching the pattern. I think we would need to split and aggregate responses that pas through the various response processor chains.

Describe alternatives you've considered
To avoid the conflict scenario described above, if no search pipeline is specified, we could simply refuse to execute search requests across an index pattern where more than one index-default search pipeline is defined.

(As a note, currently, if a search pipeline is specified in the search request, via the ?search_pipeline=... param, we use that, regardless of its applicability to the index.)

Additional context
Another possible challenge is that I've also suggested on #11053 that index aliases should be able to specify a search pipeline. If we go through with that, then what should run? Should the alias pipeline override the index default pipeline? Should we try to compose them, such that the alias request processors run, then the index default request processors run, then the query + fetch executes, then the index default response processors run, then the alias response processors run?

[msfroh]

I really like the response from @peternied on #11053, pointing to #6181.

I think I'm going to comment over there, since I think the idea of a "view" that is able to point to multiple indices (probably an index pattern?) and apply a search pipeline, and be subject to access controls could address this.

Hopefully, we'll be able to close this one out and track the work over there.

[msfroh]

Oh -- actually, I forgot that I created #7512, so this is already essentially a duplicate.