Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate _source_include and _source_exclude url parameters #33475

Merged
merged 5 commits into from
Oct 29, 2018
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1327,10 +1327,10 @@ Params withFetchSourceContext(FetchSourceContext fetchSourceContext) {
putParam("_source", Boolean.FALSE.toString());
}
if (fetchSourceContext.includes() != null && fetchSourceContext.includes().length > 0) {
putParam("_source_include", String.join(",", fetchSourceContext.includes()));
putParam("_source_includes", String.join(",", fetchSourceContext.includes()));
}
if (fetchSourceContext.excludes() != null && fetchSourceContext.excludes().length > 0) {
putParam("_source_exclude", String.join(",", fetchSourceContext.excludes()));
putParam("_source_excludes", String.join(",", fetchSourceContext.excludes()));
}
}
return this;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2713,13 +2713,13 @@ private static void randomizeFetchSourceContextParams(Consumer<FetchSourceContex
String[] includes = new String[numIncludes];
String includesParam = randomFields(includes);
if (numIncludes > 0) {
expectedParams.put("_source_include", includesParam);
expectedParams.put("_source_includes", includesParam);
}
int numExcludes = randomIntBetween(0, 5);
String[] excludes = new String[numExcludes];
String excludesParam = randomFields(excludes);
if (numExcludes > 0) {
expectedParams.put("_source_exclude", excludesParam);
expectedParams.put("_source_excludes", excludesParam);
}
consumer.accept(new FetchSourceContext(true, includes, excludes));
}
Expand Down
8 changes: 4 additions & 4 deletions docs/reference/docs/get.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -73,14 +73,14 @@ GET twitter/_doc/0?_source=false
// CONSOLE
// TEST[setup:twitter]

If you only need one or two fields from the complete `_source`, you can use the `_source_include`
& `_source_exclude` parameters to include or filter out that parts you need. This can be especially helpful
If you only need one or two fields from the complete `_source`, you can use the `_source_includes`
& `_source_excludes` parameters to include or filter out that parts you need. This can be especially helpful
with large documents where partial retrieval can save on network overhead. Both parameters take a comma separated list
of fields or wildcard expressions. Example:

[source,js]
--------------------------------------------------
GET twitter/_doc/0?_source_include=*.id&_source_exclude=entities
GET twitter/_doc/0?_source_includes=*.id&_source_excludes=entities
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]
Expand Down Expand Up @@ -232,7 +232,7 @@ You can also use the same source filtering parameters to control which parts of

[source,js]
--------------------------------------------------
GET twitter/_doc/1/_source?_source_include=*.id&_source_exclude=entities'
GET twitter/_doc/1/_source?_source_includes=*.id&_source_excludes=entities'
--------------------------------------------------
// CONSOLE
// TEST[continued]
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/docs/multi-get.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ GET /test/type/_mget
By default, the `_source` field will be returned for every document (if stored).
Similar to the <<get-source-filtering,get>> API, you can retrieve only parts of
the `_source` (or not at all) by using the `_source` parameter. You can also use
the url parameters `_source`,`_source_include` & `_source_exclude` to specify defaults,
the url parameters `_source`,`_source_includes` & `_source_excludes` to specify defaults,
which will be used when there are no per-document instructions.

For example:
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/search/explain.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ This will yield the same result as the previous request.
`_source`::

Set to `true` to retrieve the `_source` of the document explained. You can also
retrieve part of the document by using `_source_include` & `_source_exclude` (see <<get-source-filtering,Get API>> for more details)
retrieve part of the document by using `_source_includes` & `_source_excludes` (see <<get-source-filtering,Get API>> for more details)

`stored_fields`::
Allows to control which stored fields to return as part of the
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/search/uri-request.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ providing text to a numeric field) to be ignored. Defaults to false.
hits was computed.

|`_source`|Set to `false` to disable retrieval of the `_source` field. You can also retrieve
part of the document by using `_source_include` & `_source_exclude` (see the <<search-request-source-filtering, request body>>
part of the document by using `_source_includes` & `_source_excludes` (see the <<search-request-source-filtering, request body>>
documentation for more details)

|`stored_fields` |The selective stored fields of the document to return for each hit,
Expand Down
4 changes: 2 additions & 2 deletions rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,11 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or default list of fields to return, can be overridden on each sub-request"
},
"_source_exclude": {
"_source_excludes": {
"type" : "list",
"description" : "Default list of fields to exclude from the returned _source field, can be overridden on each sub-request"
},
"_source_include": {
"_source_includes": {
"type" : "list",
"description" : "Default list of fields to extract and return from the _source field, can be overridden on each sub-request"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,11 +102,11 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or a list of fields to return"
},
"_source_exclude": {
"_source_excludes": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
},
"_source_include": {
"_source_includes": {
"type" : "list",
"description" : "A list of fields to extract and return from the _source field"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,11 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or a list of fields to return"
},
"_source_exclude": {
"_source_excludes": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
},
"_source_include": {
"_source_includes": {
"type" : "list",
"description" : "A list of fields to extract and return from the _source field"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,11 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or a list of fields to return"
},
"_source_exclude": {
"_source_excludes": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
},
"_source_include": {
"_source_includes": {
"type" : "list",
"description" : "A list of fields to extract and return from the _source field"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,11 +69,11 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or a list of fields to return"
},
"_source_exclude": {
"_source_excludes": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
},
"_source_include": {
"_source_includes": {
"type" : "list",
"description" : "A list of fields to extract and return from the _source field"
}
Expand Down
8 changes: 8 additions & 0 deletions rest-api-spec/src/main/resources/rest-api-spec/api/get.json
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,14 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or a list of fields to return"
},
"_source_excludes": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
},
"_source_includes": {
"type" : "list",
"description" : "A list of fields to extract and return from the _source field"
},
"_source_exclude": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,11 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or a list of fields to return"
},
"_source_exclude": {
"_source_excludes": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
},
"_source_include": {
"_source_includes": {
"type" : "list",
"description" : "A list of fields to extract and return from the _source field"
},
Expand Down
4 changes: 2 additions & 2 deletions rest-api-spec/src/main/resources/rest-api-spec/api/mget.json
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or a list of fields to return"
},
"_source_exclude": {
"_source_excludes": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
},
"_source_include": {
"_source_includes": {
"type" : "list",
"description" : "A list of fields to extract and return from the _source field"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,11 +105,11 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or a list of fields to return"
},
"_source_exclude": {
"_source_excludes": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
},
"_source_include": {
"_source_includes": {
"type" : "list",
"description" : "A list of fields to extract and return from the _source field"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or a list of fields to return"
},
"_source_exclude": {
"_source_excludes": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
},
"_source_include": {
"_source_includes": {
"type" : "list",
"description" : "A list of fields to extract and return from the _source field"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,11 @@
"type" : "list",
"description" : "True or false to return the _source field or not, or a list of fields to return"
},
"_source_exclude": {
"_source_excludes": {
"type" : "list",
"description" : "A list of fields to exclude from the returned _source field"
},
"_source_include": {
"_source_includes": {
"type" : "list",
"description" : "A list of fields to extract and return from the _source field"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@
bulk:
index: test_index
type: test_type
_source_include: foo
_source_includes: foo
body: |
{ "update": { "_id": "test_id_3" } }
{ "doc": { "foo": "garply" } }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,18 +27,18 @@
- is_false: get._source.include.field2

- do:
explain: { index: test_1, type: test, id: 1, _source_include: include.field1, body: { query: { match_all: {}} } }
explain: { index: test_1, type: test, id: 1, _source_includes: include.field1, body: { query: { match_all: {}} } }
- match: { get._source.include.field1: v1 }
- is_false: get._source.include.field2

- do:
explain: { index: test_1, type: test, id: 1, _source_include: "include.field1,include.field2", body: { query: { match_all: {}} } }
explain: { index: test_1, type: test, id: 1, _source_includes: "include.field1,include.field2", body: { query: { match_all: {}} } }
- match: { get._source.include.field1: v1 }
- match: { get._source.include.field2: v2 }
- is_false: get._source.count

- do:
explain: { index: test_1, type: test, id: 1, _source_include: include, _source_exclude: "*.field2", body: { query: { match_all: {}} } }
explain: { index: test_1, type: test, id: 1, _source_includes: include, _source_excludes: "*.field2", body: { query: { match_all: {}} } }
- match: { get._source.include.field1: v1 }
- is_false: get._source.include.field2
- is_false: get._source.count
Original file line number Diff line number Diff line change
Expand Up @@ -36,18 +36,18 @@
- is_false: _source.include.field2

- do:
get: { index: test_1, type: test, id: 1, _source_include: include.field1 }
get: { index: test_1, type: test, id: 1, _source_includes: include.field1 }
- match: { _source.include.field1: v1 }
- is_false: _source.include.field2

- do:
get: { index: test_1, type: test, id: 1, _source_include: "include.field1,include.field2" }
get: { index: test_1, type: test, id: 1, _source_includes: "include.field1,include.field2" }
- match: { _source.include.field1: v1 }
- match: { _source.include.field2: v2 }
- is_false: _source.count

- do:
get: { index: test_1, type: test, id: 1, _source_include: include, _source_exclude: "*.field2" }
get: { index: test_1, type: test, id: 1, _source_includes: include, _source_excludes: "*.field2" }
- match: { _source.include.field1: v1 }
- is_false: _source.include.field2
- is_false: _source.count
Expand All @@ -66,3 +66,37 @@
- match: { _id: "1" }
- match: { fields.count: [1] }
- match: { _source.include.field1: v1 }

---
"Deprecated _source_include and _source_exclude":

- skip:
version: " - 6.4.99"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// TODO consider changing the version (if needed) once the clients specs are changed to _source_includes & _source_excludes

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd use 6.99.99 for now. After I merge this I'll backport it to the 6.x branch. Then I'll come in and update the range here to include the backport target version.

I'm fairly sure this is what caused the build to fail. If you crack open the PR build's full console it'll have a REPRODUCE WITH line that should reproduce the failure. And I'm fairly sure switching this to 6.99.99 will make it go away.

reason: _source_include and _source_exclude are deprecated from 6.5.0
features: "warnings"

- do:
indices.create:
index: test_1
body:
mappings:
test:
properties:
count:
type: integer
store: true

- do:
index:
index: test_1
type: test
id: 1
body: { "include": { "field1": "v1", "field2": "v2" }, "count": 1 }
- do:
get: { index: test_1, type: test, id: 1, _source_include: include.field1 }
warnings:
- "Deprecated parameter [_source_include] used, expected [_source_includes] instead"
- do:
get: { index: test_1, type: test, id: 1, _source_includes: include, _source_exclude: "*.field2" }
warnings:
- "Deprecated parameter [_source_exclude] used, expected [_source_excludes] instead"
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,18 @@
body: { "include": { "field1": "v1", "field2": "v2" }, "count": 1 }

- do:
get_source: { index: test_1, type: test, id: 1, _source_include: include.field1 }
get_source: { index: test_1, type: test, id: 1, _source_includes: include.field1 }
- match: { include.field1: v1 }
- is_false: include.field2

- do:
get_source: { index: test_1, type: test, id: 1, _source_include: "include.field1,include.field2" }
get_source: { index: test_1, type: test, id: 1, _source_includes: "include.field1,include.field2" }
- match: { include.field1: v1 }
- match: { include.field2: v2 }
- is_false: count

- do:
get_source: { index: test_1, type: test, id: 1, _source_include: include, _source_exclude: "*.field2" }
get_source: { index: test_1, type: test, id: 1, _source_includes: include, _source_excludes: "*.field2" }
- match: { include.field1: v1 }
- is_false: include.field2
- is_false: count
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ setup:

- do:
mget:
_source_include: "include.field1,count"
_source_includes: "include.field1,count"
index: test_1
body: { ids: [ 1,2 ] }
- match: { docs.0._source: { include: { field1: v1 }, count: 1} }
Expand All @@ -111,8 +111,8 @@ setup:

- do:
mget:
_source_include: include
_source_exclude: "*.field2"
_source_includes: include
_source_excludes: "*.field2"
index: test_1
body: { ids: [ 1,2 ] }
- match: { docs.0._source: { include: { field1: v1 } } }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,20 +49,20 @@ setup:
- is_false: hits.hits.0._source.include.field2

---
"_source include and _source in body":
- do: { search: { _source_include: include.field1, body: { _source: include.field2, query: { match_all: {} } } } }
"_source_includes and _source in body":
- do: { search: { _source_includes: include.field1, body: { _source: include.field2, query: { match_all: {} } } } }
- match: { hits.hits.0._source.include.field1: v1 }
- is_false: hits.hits.0._source.include.field2

---
"_source_include":
- do: { search: { _source_include: include.field1, body: { query: { match_all: {} } } } }
"_source_includes":
- do: { search: { _source_includes: include.field1, body: { query: { match_all: {} } } } }
- match: { hits.hits.0._source.include.field1: v1 }
- is_false: hits.hits.0._source.include.field2

---
"_source_exclude":
- do: { search: { _source_exclude: count, body: { query: { match_all: {} } } } }
"_source_excludes":
- do: { search: { _source_excludes: count, body: { query: { match_all: {} } } } }
- match: { hits.hits.0._source.include: { field1 : v1 , field2: v2 }}
- is_false: hits.hits.0._source.count

Expand Down
Loading