REST API compatiblity documentation (elastic#83487)

Co-authored-by: debadair <[email protected]>

docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc

@@ -18,6 +18,11 @@ is replaced with a new endpoint that does not contain `_xpack`. As an example,
 *Impact* +
 Use the replacement REST API endpoints. Requests submitted to the `_xpack`
 API endpoints will return an error.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>,
+any requests that include the`_xpack` prefix are rerouted to the corresponding
+URL without the `_xpack` prefix.
 ====
 
 [[remove-mapping-type-api-endpoints]]
@@ -161,6 +166,12 @@ also been removed. Use a typeless endpoint instead.
 *Impact* +
 Update your application to use typeless REST API endpoints. Requests to
 endpoints that contain a mapping type will return an error.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>, if a request includes a custom mapping type it is ignored. The request is rerouted to the corresponding typeless URL.
+Custom mapping types in request bodies and type related HTTP
+parameters are ignored, and responses, where warranted, include `_type` : `_doc`.
+
 ====
 
 .{ccs-cap} ({ccs-init}) is now only backward-compatible with the previous minor version.
@@ -214,6 +225,10 @@ sort buckets by their term, use `_key` instead.
 *Impact* +
 Discontinue use of the `_term` order key. Requests that include a `_term` order
 key will return an error.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>,
+the `_term` order is ignored and `key` is used instead.
 ====
 
 [[remove-time-order-key]]
@@ -227,6 +242,10 @@ values. To sort buckets by their key, use `_key` instead.
 *Impact* +
 Discontinue use of the `_time` order key. Requests that include a `_time` order
 key will return an error.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>,
+the `_time` order is ignored and `_key` is used instead.
 ====
 
 [[remove-moving-avg-agg]]
@@ -242,6 +261,8 @@ aggregation] instead.
 *Impact* +
 Discontinue use of the `moving_avg` aggregation. Requests that include the
 `moving_avg` aggregation will return an error.
+
+
 ====
 
 [[percentile-duplication]]
@@ -257,6 +278,7 @@ its values must be unique. Otherwise, an exception occurs.
 Use unique values in the `percents` parameter of the `percentiles` aggregation.
 Requests containing duplicate values in the `percents` parameter will return
 an error.
+
 ====
 
 [[date-histogram-interval]]
@@ -276,6 +298,10 @@ Uses of the `interval` parameter in either the `date_histogram` aggregation or
 the `date_histogram` composite source will now generate an error.  Instead
 please use the more specific `fixed_interval` or `calendar_interval`
 parameters.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>,
+the `interval` parameter is adapted to either a fixed or calendar interval.
 ====
 
 [[ngram-edgengram-filter-names-removed]]
@@ -436,6 +462,11 @@ parameter is now removed in 8.0.
 Use the {ref}/indices-templates-v1.html[create or update index template API]'s
 `index_patterns` parameter. Requests that include the `template` parameter will
 return an error.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>,
+the `template` parameter is mapped to `index_patterns`.
+
 ====
 
 .Synced flush has been removed.
@@ -448,6 +479,10 @@ instead as it has the same effect as a synced flush in 7.6 and later.
 *Impact* +
 Use the {ref}/indices-flush.html[flush API]. Requests to the
 `/<index>/flush/synced` or `/flush/synced` endpoints will return an error.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>,
+the request to synced flush is routed to the equivalent non-synced flush URL.
 ====
 
 .The default for the `?wait_for_active_shards` parameter on the close index API has changed.
@@ -477,6 +512,10 @@ stats for specific mapping types. Mapping types have been removed in 8.0.
 *Impact* +
 Discontinue use of the `types` query parameter. Requests that include the
 parameter will return an error.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>,
+the `types` query parameter is ignored and stats are returned for the entire index.
 ====
 
 .The `user_agent` ingest processor's `ecs` parameter has no effect.
@@ -504,6 +543,11 @@ include a mapping type name. Mapping types have been removed in 8.x.
 *Impact* +
 Discontinue use of the `include_type_name` query parameter. Requests that
 include the parameter will return an error.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>,
+the `include_type_name` query parameter is ignored and any custom mapping types
+in the request are removed.
 ====
 
 .Reindex from remote now re-encodes URL-encoded index names.
@@ -537,6 +581,11 @@ Similarly, the `size` parameter has been renamed to `max_docs` for
 *Impact* +
 Use the replacement parameters. Requests containing the `size` parameter will
 return an error.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>,
+the `size` parameter is mapped to the `max_docs` parameter.
+
 ====
 
 .The update by query API now rejects unsupported `script` fields.
@@ -875,6 +924,11 @@ removed in 8.0.
 *Impact* +
 Discontinue use of the `use_field_mapping` request body parameter. Requests
 containing this parameter will return an error.
+
+*Compatibility* +
+When <<rest-api-compatibility>> is <<request-rest-api-compatibility,requested>>,
+the `use_field_mapping` parameter is ignored.
+
 ====
 
 .The search API's `from` request body and url parameter cannot be negative.
@@ -929,6 +983,7 @@ The `type` query has been removed. Mapping types have been removed in 8.0.
 *Impact* +
 Discontinue use of the `type` query. Requests that include the `type` query
 will return an error.
+
 ====
 
 .The `kibana_user` role has been renamed `kibana_admin`.

docs/reference/rest-api/index.asciidoc

@@ -12,6 +12,7 @@ not be included yet.
 
 * <<api-conventions, API conventions>>
 * <<common-options, Common Options>>
+* <<rest-api-compatibility, REST API Compatibility>>
 * <<autoscaling-apis, Autoscaling APIs>>
 * <<cat, cat APIs>>
 * <<cluster, Cluster APIs>>
@@ -52,6 +53,7 @@ not be included yet.
 
 include::{es-repo-dir}/api-conventions.asciidoc[]
 include::{es-repo-dir}/rest-api/common-options.asciidoc[]
+include::{es-repo-dir}/rest-api/rest-api-compatibility.asciidoc[]
 include::{es-repo-dir}/autoscaling/apis/autoscaling-apis.asciidoc[]
 include::{es-repo-dir}/cat.asciidoc[]
 include::{es-repo-dir}/cluster.asciidoc[]

docs/reference/rest-api/rest-api-compatibility.asciidoc

@@ -0,0 +1,96 @@
+[[rest-api-compatibility]]
+== REST API compatibility
+
+To help REST clients mitigate the impact of non-compatible (breaking)
+API changes, {es} provides a per-request, opt-in API compatibility mode.
+
+{es} REST APIs are generally stable across versions. However, some
+improvements require changes that are not compatible with previous versions.
+For example, {es} 7.x supported custom mapping types in many URL paths,
+but {es} 8.0+ does not (see <<removal-of-types>>). Specifying a custom type
+in a request sent to {es} 8.0+ returns an error. However, if you request
+REST API compatibility, {es} accepts the request even though mapping types
+are no longer supported.
+
+When an API is targeted for removal or is going to be changed in a
+non-compatible way, the original API is deprecated for one or more releases.
+Using the original API triggers a deprecation warning in the logs.
+This enables you to review the deprecation logs  and take the appropriate actions
+before upgrading. However, in some cases it is difficult to
+identify all places where deprecated APIs are being used. This is where REST API
+compatibility can help.
+
+When you request REST API compatibility, {es} attempts to honor the previous
+REST API version. {es} attempts to apply the most compatible URL, request body,
+response body, and HTTP parameters.
+
+For compatible APIs, this has no effect--it only impacts calls to APIs
+that have breaking changes from the previous version. An error can still be
+returned in compatibility mode if {es} cannot automatically resolve the incompatibilities.
+
+IMPORTANT: REST API compatibility does not guarantee the same behavior
+as the prior version. It instructs {es} to automatically resolve any
+incompatibilities so the request can be processed instead of returning an error.
+
+
+REST API compatibility should be a bridge to smooth out the upgrade process,
+not a long term strategy. REST API compatibility is only honored across one
+major version: honor 7.x requests/responses from 8.x.
+
+When you submit requests using REST API compatibility and {es} resolves
+the incompatibility, a message is written to the deprecation log with
+the category "compatible_api". Review the deprecation log to identify
+any gaps in usage and fully supported features.
+
+
+For information about specific breaking changes and the impact of requesting
+compatibility mode, see <<breaking_80_rest_api_changes, REST API changes>>
+in the migration guide.
+
+[discrete]
+[[request-rest-api-compatibility]]
+=== Requesting REST API compatibility
+
+REST API compatibility is implemented per request via the Accept
+and/or Content-Type headers.
+
+For example:
+
+[source, text]
+------------------------------------------------------------
+Accept: "application/vnd.elasticsearch+json;compatible-with=7"
+Content-Type: "application/vnd.elasticsearch+json;compatible-with=7"
+------------------------------------------------------------
+
+The Accept header is always required and the Content-Type header is
+only required when a body is sent with the request. The following values are
+valid when communicating with a 7.x or 8.x {es} server:
+[source, text]
+------------------------------------------------------------
+"application/vnd.elasticsearch+json;compatible-with=7"
+"application/vnd.elasticsearch+yaml;compatible-with=7"
+"application/vnd.elasticsearch+smile;compatible-with=7"
+"application/vnd.elasticsearch+cbor;compatible-with=7"
+------------------------------------------------------------
+The https://www.elastic.co/guide/en/elasticsearch/client/index.html[officially supported {es} clients]
+can enable REST API compatibility for all requests.
+
+To enable REST API compatibility for all requests received
+by {es} set the environment variable `ELASTIC_CLIENT_APIVERSIONING` to true.
+
+[discrete]
+=== REST API compatibility workflow
+
+To leverage REST API compatibility during an upgrade from 7.17 to {version}:
+
+1. Upgrade your https://www.elastic.co/guide/en/elasticsearch/client/index.html[{es} clients]
+to the latest 7.x version and enable REST API compatibility.
+2. Use the {kibana-ref}/upgrade-assistant.html[Upgrade Assistant]
+to review all critical issues and explore the deprecation logs.
+Some critical issues might be mitigated by REST API compatibility.
+3. Resolve all critical issues before proceeding with the upgrade.
+4. Upgrade Elasticsearch to {version}.
+5. Review the deprecation logs for entries with the category `compatible_api`.
+Review the workflow associated with the requests that relied on compatibility mode.
+6. Upgrade your {es} clients to 8.x and resolve compatibility issues manually where needed.
+

docs/reference/upgrade.asciidoc

@@ -107,6 +107,14 @@ When upgrading to a new version of {es}, you need to upgrade each
 of the products in your Elastic Stack. For more information, see the
 {stack-ref}/upgrading-elastic-stack.html[Elastic Stack Installation and Upgrade Guide].
 
+[discrete]
+[[upgrade-rest-api-compatibility]]
+=== REST API compatibility
+
+REST API compatibility in a per-request opt-in feature that can help REST clients
+mitigate non compatible (breaking) changes to the REST API.
+See <<rest-api-compatibility>> for additional information.
+
 [discrete]
 [[upgrade-fips-java17]]
 === FIPS Compliance and Java 17