open telemetry can't ingest dynatrace from aws lambda function with AWS Distro for OpenTelemetry

[eulisesquidel]

We use AWS Distro for OpenTelemetry in lambda function:
arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-5-0:1

We did a basic aws lambda function in nodejs:

exports.handler = async (event) => {
let response
console.log('[START] - OpenTelemetry / Dynatrace');
try {
response = await axios({
method: 'post', url:url
});
console.log('**************', response?.data)
} catch (error) {
console.log('ERR: ', error)
}
console.log('[END] - OpenTelemetry / Dynatrace');
return response?.data;
}

Our collector.yaml config is:

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:55680
http:
endpoint: 0.0.0.0:55681
exporters:
otlphttp:
endpoint: "https://ico19512.live.dynatrace.com/api/v2/otlp/v1/traces"
headers: {Authorization: "Api-Token dt0c01......"}
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlphttp]

Our enviroments variables config:

Our lamda run ok, but we don't see ingested nothing.

AWS Lambda function response was ok: "34.237.128.180"

Function Logs

START RequestId: b077aa35-b4bc-4604-8e5a-c67dbfa454ac Version: $LATEST
EXTENSION Name: collector State: Ready Events: [INVOKE,SHUTDOWN]
{"level":"debug","msg":"Received ","event :":"{\n\t"eventType": "INVOKE",\n\t"deadlineMs": 1660236861439,\n\t"requestId": "b077aa35-b4bc-4604-8e5a-c67dbfa454ac",\n\t"invokedFunctionArn": "arn:aws:lambda:us-east-1:658913498668:function:dyna-otel",\n\t"tracing": {\n\t\t"type": "X-Amzn-Trace-Id",\n\t\t"value": "Root=1-62f5342f-1b4ce6bd6c1ebf2c6601dc57;Parent=29fc70b85a8376fd;Sampled=0"\n\t}\n}"}
{"level":"debug","msg":"Waiting for event..."}
2022-08-11T16:54:09.465Z b077aa35-b4bc-4604-8e5a-c67dbfa454ac INFO [START] - OpenTelemetry / Dynatrace
2022-08-11T16:54:09.806Z b077aa35-b4bc-4604-8e5a-c67dbfa454ac INFO ************** 34.237.128.180

2022-08-11T16:54:09.806Z b077aa35-b4bc-4604-8e5a-c67dbfa454ac INFO [END] - OpenTelemetry / Dynatrace
END RequestId: b077aa35-b4bc-4604-8e5a-c67dbfa454ac
REPORT RequestId: b077aa35-b4bc-4604-8e5a-c67dbfa454ac Duration: 708.69 ms Billed Duration: 709 ms Memory Size: 128 MB Max Memory Used: 126 MB Init Duration: 1324.55 ms

[dyladan]

Can you verify if the spans are making it to the collector by trying to use a console exporter or something along those lines?

[jspaleta]

are there good instructions somewhere on diagnosing if a receiver ingest is working.
Its easy to assume its some other step in a pipeline thats not working, when its really something wrong with receiver ingest.

[dyladan]

I usually just try another simpler exporter (like console)

[eulisesquidel]

Thanks for all your replies.
I don't know how to use console exporter in AWS lambda function.
I will appreciate if you have a minimal example on this topic.
About spans, we try to do a minimal trace example, a nodejs client app that call our lambda function. the nodejs client app ingest ok in our dynatrace server.

Node JS Client App

Our Dynatrace

[evan-bradley]

To use the logging exporter, which logs to the console, update your Collector configuration to include it in the list of exporters so that it looks something like this:

pipelines:
  traces:
    receivers: [otlp]
    exporters: [logging, otlphttp]

This will cause the Collector to print logs for each trace received; if no logs appear, the Collector is not receiving them.

Looking at your configuration, one thing I see is that the Dynatrace endpoint in your Collector configuration includes the /v1/traces path, which the OTLPHTTP exporter should add automatically when exporting traces. You can see an example configuration in the Dynatrace documentation for what this might look like. In particular, the exporter config should look something like below, which is reproduced from the documentation page I linked:

exporters:
  otlphttp:
    endpoint: "https://{your-environment-id}.live.dynatrace.com/api/v2/otlp"
    headers:
      Authorization: "Api-Token <API_TOKEN>"

[eulisesquidel]

Hi Evan,
Thanks for your answer.
Let's try it with your suggestions. I will comment soon.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.