[libbeat] add_cloud_metadata - startup blocked by AWS IMSDv2 token fetch

[andrewkroh]

The AWS provider in the add_cloud_metadata processor was modified to support IMSDv2 in #28285. There are two issues affecting the implementation.

1. Fetching of the IMSDv2 token blocks the startup of the processor. The processor was designed to try fetching metadata from all enabled providers (e.g. aws, gcp, openstack) in parallel and asynchronously from the main thread. This allows the Beat to startup without being blocked on Cloud provider detection logic.
   
   The implementation fetches the IMSDv2 token in a non-async manner such that it blocks startup and does not happen in parallel with all other provider implementations. In non-AWS environments the Beat blocks until a HTTP timeout is reached (default is 3s).
   
   The processor needs to be modified to fetch the IMSDv2 token asynchronously. Some changes might be needed to allow chaining the request to fetch the token and use the token in the AWS metadata service requests.

2. (update: this should be fixed by [add_cloud_metadata] Remove logger for AWS/EC2 #36829) It logs warnings for something that is expected to fail when the Beat runs outside of AWS. This should not be logged as a warning. It should be handled in the same manner as a non-response is handled by all other provider implementations. If the error is returned in the result from the provider then the processor will log the disposition at the debug level.

{"log.level":"warn","@timestamp":"2022-09-12T17:09:41.120-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}

[andrewkroh]

These two issues appear related to bullet (2) above.

Related:

• add_cloud_metadata produces errors running in non-cloud elastic-agent install #36018
• [libbeat] add_cloud_metadata processor logs confusing warnings for AWS even when running on other CSPs #35147