(feature) Amazon Kinesis Exporter

[thomaspoignant]

Requirements

GO Feature Flag is supporting different solution to export the flag evaluation to your existing system.

The list of existing exporter destination are available here.

GO Feature Flag is already supporting some queue systems (SQS, SNS, Kafka, GCP PubSub ...), but adding the compatibility to AWS Kinesis would be a very nice addition for the users using it in their AWS instance.

A good example on how to implement this exporter is to look at the Kafka exporter https://github.com/thomaspoignant/go-feature-flag/pull/1426/files.

[baitcode]

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

https://www.linkedin.com/in/batiyiv/
7 years of experience with AWS

How I plan on tackling this issue

I'll check structure of Kafka explorer, and recreate similar. Will create a Kinesis stream for test, manually test against it. Maybe will use localstack to organise testing.

[NueloSE]

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

I am a software developer with experience in building frontend application which are web and mobile compactable. Experienced in some of the tech stack used like typescript JS python C reactnative But am open to learning with the project

How I plan on tackling this issue

A psuedo approach to solving the above task would be :

Step 1: Analyze the Existing Structure
I'll carefully examine how the Kafka exporter is structured. This will involve looking at the KafkaExporter struct, how it implements the Exporter interface, and how it uses the github.com/confluentinc/kafka-go package for Kafka integration.

Step 2: Design the Kinesis Exporter
Next, I'll create an AWSKinesisExporter struct and implement the Exporter interface for it. I'll also decide which AWS SDK version to use - likely the latest AWS SDK v2 for Kinesis.

Step 3: Implement the Exporter Interface
I'll make sure to implement all required methods of the Exporter interface in my AWSKinesisExporter. This will include the Export method, which will handle sending events to Kinesis.

Step 4: Set Up AWS Credentials
Before I can interact with AWS services, I need to ensure that proper AWS credentials are set up in my Go environment. This might involve configuring environment variables or using AWS credentials file.

Step 5: Create a Kinesis Client
Using the AWS SDK, I'll create a Kinesis client. This involves initializing the client with the necessary configuration.

Step 6: Implement the Export Method
In the Export method, I'll convert the incoming event to a format suitable for Kinesis and then send it to the designated stream.

Step 7: Handle Errors and Logging
Throughout the implementation, I'll make sure to implement proper error handling and logging. This will help with debugging and monitoring the exporter's behavior.

Step 8: Test the Implementation
Finally, I'll create comprehensive unit tests to verify that my Kinesis exporter works correctly. This will involve simulating various scenarios and edge cases.

[baitcode]

@thomaspoignant
I've implemented most of it. Couple of questions to finalise:

1. What does isBulk() mean? I see that I get array of events as an input, and kinesis stream allows for sending multiple events, isn't that bulk?
2. Kinesis event has to be 1 Mb max, is that ok to check events that are bigger than 1 Mb, skip those for sending, and logging information about that event (not the event itself, of course).

[thomaspoignant]

@baitcode How super nice to see that you progressed that fast.

What does isBulk() mean? I see that I get array of events as an input, and kinesis stream allows for sending multiple events, isn't that bulk?

isBulk() is a helper function to know if we should buffer the data before sending them or not.
In your case since Kinesis is a queue system we should just return false here, we buffer when we write files like for S3 for example.

Kinesis event has to be 1 Mb max, is that ok to check events that are bigger than 1 Mb, skip those for sending, and logging information about that event (not the event itself, of course).

Since we are not in bulk mode here (because of the answer from the previous question) I doubt we will reach the 1Mb event size, so your approach is ok for me.

[baitcode]

@thomaspoignant Is order of the events is a stream important? I'm thinking if I should distribute events into different Kinesis shards by default or just use a single hardcoded shard. I'll do the single shard for now, and let's fix it on review phase if you feel like it's a wrong approach.

[baitcode]

@thomaspoignant I'll do another round of review later today, but would love if you did as well on your side. I don't expect any major changes except for documentation, as I tested documentation poorly.