New serverless pattern - kinesis-lambda-dynamodb-pipeline-dotnet-cdk

[tejasavora]

Issue #, if available:
N/A

Description of changes:
This pull request implements a new serverless pattern demonstrating a real-time data pipeline using Amazon Kinesis, AWS Lambda, Amazon DynamoDB, and Amazon SQS. The key changes include:

1. Created a new CDK stack (KinesisLambdaDynamoDbCdkStack) that sets up:

   • An Amazon Kinesis Data Stream for data ingestion
   • An AWS Lambda function for data processing
   • Two Amazon DynamoDB tables: one for processed data and another for error logging
   • An Amazon SQS queue as a Dead Letter Queue (DLQ) for handling persistently failing records

2. Implemented a Lambda function (DataProcessFunction) that:

   • Processes records from the Kinesis stream
   • Stores successfully processed data in DynamoDB
   • Logs processing errors to a separate DynamoDB table

3. Configured robust error handling:

   • Set up retry attempts for failed record processing
   • Implemented automatic forwarding of consistently failing records to the DLQ

4. Added comprehensive testing instructions, including:

   • Steps for testing with AWS CLI and console
   • Information on running local unit and integration tests
   • Suggestions for end-to-end testing using AWS Step Functions

5. Updated README.md with detailed information about the pattern, its architecture, deployment instructions, and testing procedures.

This pattern showcases best practices for building scalable, resilient serverless data processing pipelines using AWS services and the AWS CDK with .NET.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.