AWS DataDog Forwarder failing due to old URLlib3

[rknechtel]

Describe what happened:
Using latest release of AWS DataDog Forwarder Lambda code, all lambda calls are failing due to the use of an old urllib3 version.

Log Group: /aws/lambda/DatadogIntegration-ForwarderStack-H16PMQ-Forwarder-PYitVeIgHSnt

Errors in my CloudWatch logs

START RequestId: 21fd84d1-1a80-44c3-9ca6-67d64f4f65c4 Version: $LATEST
LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'OpenSSL 1.0.2k-fips  26 Jan 2017'. See: https://github.com/urllib3/urllib3/issues/2168
Traceback (most recent call last):
END RequestId: 21fd84d1-1a80-44c3-9ca6-67d64f4f65c4

When trying to create Lambda code from source when I run pip3 install -r requirements.txt
I see this error:
ERROR: opentelemetry-api 1.27.0 has requirement importlib-metadata<=8.4.0,>=6.0, but you'll have importlib-metadata 8.5.0 which is incompatible.
ERROR: datadog-lambda 5.87.0 has requirement urllib3<2.0.0; python_version < "3.11", but you'll have urllib3 2.2.3 which is incompatible.

The requirements.txt has these:
datadog-lambda==5.87.0
urllib3>=1.26.19,<3.0

The urlib3 allows versions up to 3.0 but the datadog-lambda version only allows urllib versions up to 2.0 - which is where the CloudWatch log errors are coming from (urllib3 2.0).

I made updates to the requirements.txt locally and then built the code. I made the following changes:
Latest version of datadog-lambda: 6.98.0

Original in requirements.txt:
datadog-lambda==5.87.0
ddtrace==2.6.5

Updated:
datadog-lambda==6.98.0
ddtrace>=2.10.0

It will build then. But when uploading that code to the Lambda I still get the errors.

Describe what you expected:
Lambda should work with a new urllib3 version.

Steps to reproduce the issue:
deploy latest release code (aws-dd-forwarder-3.123.0.zip) to DataDog AWS lambda (DatadogIntegration-ForwarderStack-H16PMQ-Forwarder-PYitVeIgHSnt).

Look in CloudWatch logs for error.

[rknechtel]

I got it working here is what I did:
Updated to latest CloudFormation Template:
https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml

git cloned the repo.
Updated the requirements.txt:
Original in requirements.txt:
datadog-lambda==5.87.0
ddtrace==2.6.5

Updated:
datadog-lambda==6.98.0
ddtrace>=2.10.0

Built lambda code:
python3.11 -m venv ~/projects/DataDog/datadog-serverless-functions/aws/logs_monitoring/v-env
source ~/projects/DataDog/datadog-serverless-functions/aws/logs_monitoring/v-env/bin/activate
~/projects/DataDog/datadog-serverless-functions/aws/logs_monitoring/v-env/bin/pip3.11 install -r requirements.txt

Copied:
init.py
customized_log_group.py
enhanced_lambda_metrics.py
forwarder.py
lambda_function.py
requirements.txt
settings.py
setup.py
telemetry.py
template.yaml

To:
/projects/DataDog/datadog-serverless-functions/aws/logs_monitoring/v-env/lib/python3.11/site-packages

Then zipped up everything in that directory and deployed it to the lambda. Now my CloudWatch Logs are getting to DataDog Log Explorer.