[8.15](backport #40512) [ES output] Correctly log event fields in events log file

[mergify]

Proposed commit message

The Elasticsearch output, when faced with ingestion errors was logging the raw publisher.Event that had already been encoded, hence no event fields were present in the logs.

This commit fixes it by adding a String method to the encodedEvent type and using the encodedEvent in the logs instead of the publisher.Event.

Closes: #40509

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None, it fixes a bug.

## Author's Checklist

How to test this PR locally

Start Filebeat with the following configuration

filebeat.inputs:
  - type: filestream
    id: filestream-input-id
    enabled: true
    parsers:
      - ndjson:
          target: ""
          overwrite_keys: true
          expand_keys: true
          add_error_key: true
          ignore_decoding_error: false
    paths:
      - /tmp/flog.log

output:
  elasticsearch:
    hosts:
      - localhost:9200
    protocol: https
    username: elastic
    password: changeme
    allow_older_versions: true
    ssl.verification_mode: none

logging:
  level: debug
  event_data:
    files:
      name: filebeat-events-data # that's the default, change it if you want another name.

Create the log file /tmp/flog.log with the following content:

{"message":"foo bar","int":10,"string":"str"}
{"message":"another message","int":20,"string":"str2"}
{"message":"index failure","int":"not a number","string":10}
{"message":"second index failure","int":"not a number","string":10}
A broken JSON

Look for the event log file: logs/filebeat-events-data*.ndjson and ensure the logged events contains their fields.

Here is an example of how the content of the event log file should look like:

{
  "log.level": "warn",
  "@timestamp": "2024-08-13T16:42:09.008-0400",
  "log.logger": "elasticsearch",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus",
    "file.name": "elasticsearch/client.go",
    "file.line": 490
  },
  "message": "Cannot index event '{\"@timestamp\":\"2024-08-13T20:42:05.928Z\",\"host\":{\"name\":\"millennium-falcon\"},\"agent\":{\"version\":\"8.16.0\",\"ephemeral_id\":\"6d195bff-27a4-40c4-9b3e-c3ecb068f06e\",\"id\":\"6a760df8-a3e6-4369-886a-3f499c792302\",\"name\":\"millennium-falcon\",\"type\":\"filebeat\"},\"log\":{\"file\":{\"device_id\":\"40\",\"inode\":\"51817\",\"path\":\"/tmp/flog.log\"},\"offset\":101},\"string\":10,\"message\":\"index failure\",\"int\":\"not a number\",\"input\":{\"type\":\"filestream\"},\"ecs\":{\"version\":\"8.0.0\"}}\n' (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:380] failed to parse field [int] of type [long] in document with id 'iVl6TZEBA82tHj8dCPpP'. Preview of field's value: 'not a number'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"For input string: \\\"not a number\\\"\"}}, dropping event!",
  "service.name": "filebeat",
  "log.type": "event",
  "ecs.version": "1.6.0"
}
{
  "log.level": "warn",
  "@timestamp": "2024-08-13T16:42:09.009-0400",
  "log.logger": "elasticsearch",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus",
    "file.name": "elasticsearch/client.go",
    "file.line": 490
  },
  "message": "Cannot index event '{\"@timestamp\":\"2024-08-13T20:42:05.928Z\",\"host\":{\"name\":\"millennium-falcon\"},\"agent\":{\"name\":\"millennium-falcon\",\"type\":\"filebeat\",\"version\":\"8.16.0\",\"ephemeral_id\":\"6d195bff-27a4-40c4-9b3e-c3ecb068f06e\",\"id\":\"6a760df8-a3e6-4369-886a-3f499c792302\"},\"ecs\":{\"version\":\"8.0.0\"},\"log\":{\"offset\":162,\"file\":{\"path\":\"/tmp/flog.log\",\"device_id\":\"40\",\"inode\":\"51817\"}},\"message\":\"second index failure\",\"int\":\"not a number\",\"string\":10,\"input\":{\"type\":\"filestream\"}}\n' (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:401] failed to parse field [int] of type [long] in document with id 'ill6TZEBA82tHj8dCPpP'. Preview of field's value: 'not a number'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"For input string: \\\"not a number\\\"\"}}, dropping event!",
  "service.name": "filebeat",
  "log.type": "event",
  "ecs.version": "1.6.0"
}

## Use cases
## Screenshots

---

This is an automatic backport of pull request #40512 done by Mergify.

[mergify]

Cherry-pick of e7732c6 has failed:

On branch mergify/bp/8.15/pr-40512
Your branch is up to date with 'origin/8.15'.

You are currently cherry-picking commit e7732c6669.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   filebeat/tests/integration/event_log_file_test.go
	modified:   libbeat/outputs/elasticsearch/event_encoder.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/outputs/elasticsearch/client.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

buildkite test this

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏