Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(lambda-event-sources): starting position timestamp for kafka #31439

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

nikovirtala
Copy link
Contributor

@nikovirtala nikovirtala commented Sep 13, 2024

Issue # (if applicable)

Closes #31808

Reason for this change

It was impossible to start consuming a Kafka topic from a specific point in time.

Description of changes

The user may now set startingPositionTimestamp to ManagedKafkaEventSource and SelfManagedKafkaEventSource to start consuming a Kafka topic from a specific point in time.

Lambda and CloudFormation have supported the functionality for a while, and other stream event sources like KinesisEventSource already had a CDK implementation for it, so there's nothing new under the sky — just expanding it to cover sources we are using :)

Description of how you validated changes

The change is tested by adding similar unit tests to other event sources supporting this functionality.

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@aws-cdk-automation aws-cdk-automation requested a review from a team September 13, 2024 10:41
@github-actions github-actions bot added p2 valued-contributor [Pilot] contributed between 6-12 PRs to the CDK labels Sep 13, 2024
@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Sep 13, 2024
@moelasmar
Copy link
Contributor

could you please add a new GH issue as a Feature request, and link it to this issue.

Copy link
Contributor

@moelasmar moelasmar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, this is a feature and not a chore, and it needs update in Readme, and new integration test case

@aws-cdk-automation aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Sep 16, 2024
Copy link
Contributor

@sumupitchayan sumupitchayan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed with @moelasmar, this seems like a feat to me - @nikovirtala can you please add an integ test and README documentation on this?

@aws-cdk-automation
Copy link
Collaborator

This PR has been in the CHANGES REQUESTED state for 3 weeks, and looks abandoned. To keep this PR from being closed, please continue work on it. If not, it will automatically be closed in a week.

@moelasmar moelasmar changed the title chore(lambda-event-sources): starting position timestamp for kafka feat(lambda-event-sources): starting position timestamp for kafka Oct 14, 2024
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@nikovirtala nikovirtala force-pushed the chore/kafka-starting-position-timestamp branch 2 times, most recently from e521983 to ab944c3 Compare October 18, 2024 12:10
@nikovirtala
Copy link
Contributor Author

Agreed with @moelasmar, this seems like a feat to me - @nikovirtala can you please add an integ test and README documentation on this?

README is clear, but creating the integration test for this is a lot of work:

  1. there are no existing integration tests for any other Event Source Mappings
  2. testing Kafka Event Source Mapping requires creating an MSK cluster or such, which is not as trivial as creating a simple resource like DynamoDB Table (I hope there is something I can borrow from the MSK module, if one exist)

@nikovirtala
Copy link
Contributor Author

nikovirtala commented Oct 18, 2024

Exemption Request

  1. testing Kafka Event Source Mapping requires creating an MSK cluster or such, which is not as trivial as creating a simple resource like DynamoDB Table (I hope there is something I can borrow from the MSK module, if one exist)

Ok, the MSK module is in alpha state, and even that doesn't have an integration test that would create a topic and messages to the topic — which would be required to test this feature 😞

I doubt that creating all that is too much for me.

@nikovirtala nikovirtala force-pushed the chore/kafka-starting-position-timestamp branch from ab944c3 to 811d9b7 Compare October 18, 2024 13:03
@nikovirtala nikovirtala force-pushed the chore/kafka-starting-position-timestamp branch from 811d9b7 to 81e290b Compare October 18, 2024 13:07
@github-actions github-actions bot added the feature-request A feature should be added or improved. label Oct 18, 2024
@aws-cdk-automation aws-cdk-automation added the pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback. label Oct 18, 2024
@aws-cdk-automation
Copy link
Collaborator

The pull request linter fails with the following errors:

❌ Features must contain a change to an integration test file and the resulting snapshot.

PRs must pass status checks before we can provide a meaningful review.

If you would like to request an exemption from the status checks or clarification on feedback, please leave a comment on this PR containing Exemption Request and/or Clarification Request.

✅ A exemption request has been requested. Please wait for a maintainer's review.

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 81e290b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request A feature should be added or improved. p2 pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback. valued-contributor [Pilot] contributed between 6-12 PRs to the CDK
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(lambda-event-sources): (starting position timestamp for kafka)
4 participants