[8.12](backport #38116) x-pack/filebeat/input/httpjson: drop response…

… bodies at end of execution (#38128)

* x-pack/filebeat/input/httpjson: drop response bodies at end of execution (#38116)

The response bodies of the first and last responses were being held in a
closed-over variable resulting in high static memory loads in some
situations. The bodies are not used between periodic executions with the
documentation stating that only cursor values are persisted across
restarts. The difference in behaviour between using the body field over
a restart versus over a sequence of executions in the same run make them
unsafe, so clarify the persistence behaviour in the documentation and
free the bodies at the end of an execution.

A survey of integrations that use the httpjson input did not identify
any that are using behaviour that is being removed, but we will need to
keep an eye on cases that may have been missed. In general, if
persistence is being depended on, the cursor should be being used.

(cherry picked from commit 353dab3)

* remove irrelevant changelog entries

---------

Co-authored-by: Dan Kortschak <[email protected]>

CHANGELOG.next.asciidoc

@@ -47,6 +47,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 
 *Filebeat*
 
+- Prevent HTTPJSON holding response bodies between executions. {issue}35219[35219] {pull}38116[38116]
 
 *Heartbeat*
 

x-pack/filebeat/docs/inputs/input-httpjson.asciidoc

@@ -117,7 +117,7 @@ The state has the following elements:
 - `body`: A map containing the body. References the next request body when used in <<request-transforms-headers>> or <<response-pagination>> configuration sections, and to the last response body when used in <<response-transforms>> or <<response-split>> configuration sections.
 - `cursor`: A map containing any data the user configured to be stored between restarts (See <<cursor>>).
 
-All of the mentioned objects are only stored at runtime, except `cursor`, which has values that are persisted between restarts.
+All of the mentioned objects are only stored at runtime during the execution of the periodic request, except `cursor`, which has values that are persisted between periodic request and restarts.
 
 [[transforms]]
 ==== Transforms

x-pack/filebeat/input/httpjson/input.go

@@ -153,6 +153,12 @@ func run(ctx v2.Context, cfg config, pub inputcursor.Publisher, crsr *inputcurso
 	trCtx.cursor.load(crsr)
 
 	doFunc := func() error {
+		defer func() {
+			// Clear response bodies between evaluations.
+			trCtx.firstResponse.body = nil
+			trCtx.lastResponse.body = nil
+		}()
+
 		log.Info("Process another repeated request.")
 
 		startTime := time.Now()