Manage AWS SDK(s) version with Dependabot

[zmoog]

Proposed commit message

Set up Dependabot to manage the AWS SDK version.

With the current reactive and manual process, our dependencies are often outdated. To release a bugfix to a dependency, we need to wait for the following stack release instead of merging it shortly after it's available from AWS.

See #39492 to learn more.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates Keep the AWS/Azure/GCP SDK version up to date #39492

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @zmoog? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[zmoog]

The following dependencies are AWS-related, but I'm not sure they depend on the AWS SDK.

github.com/awslabs/kinesis-aggregation/go/v2
github.com/aws/smithy-go
github.com/awslabs/goformation/v4
github.com/aws/aws-lambda-go

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[zmoog]

The following dependencies are AWS-related, but I'm not sure they depend on the AWS SDK.

github.com/awslabs/kinesis-aggregation/go/v2
github.com/aws/smithy-go
github.com/awslabs/goformation/v4
github.com/aws/aws-lambda-go

github.com/awslabs/goformation/v4, github.com/aws/smithy-go, and github.com/aws/aws-lambda-go do not directly depend on the AWS SDK.

github.com/awslabs/kinesis-aggregation depends on github.com/aws/aws-sdk-go-v2/service/kinesis

[zmoog]

github.com/awslabs/kinesis-aggregation depends on github.com/aws/aws-sdk-go-v2/service/kinesis

It seems Functionbeat only uses this package to reference the types.Record struct.

[kaiyan-sheng]

It seems Functionbeat only uses this package to reference the types.Record struct.

Functionbeat is also using the deaggregator: https://github.com/elastic/beats/blob/main/x-pack/functionbeat/provider/aws/aws/transformer/transformer.go#L18

[zmoog]

It seems Functionbeat only uses this package to reference the types.Record struct.

Functionbeat is also using the deaggregator: https://github.com/elastic/beats/blob/main/x-pack/functionbeat/provider/aws/aws/transformer/transformer.go#L18

Oh, good catch!

It seems the awslabs/kinesis-aggregation/blob/master/go/deaggregator/deaggregator.go only use the github.com/aws/aws-sdk-go-v2/service/kinesis package to reference the types.Record struct.

We can probably initially focus on the AWS SDK only (github.com/aws/aws-sdk-go-v2/*), leaving other AWS-related packages for a later addition.

[zmoog]

We need to revise this PR after merging #40125

[dliappis]

LGTM (CI is green)

[zmoog]

I am waiting for #40150 to merge before rebasing on it.

[zmoog]

I rebased on main to get the changes we made for Azure and GCP.

@rowlandgeoff, would you mind taking a final look at the AWS config?

[pazone]

Not sure if it's important, but we also use aws-lambda-go here

[kruskall]

should we add github.com/aws/aws-lambda-go, github.com/aws/smithy-go and similar libs ?

[zmoog]

Yep, it makes sense! I created separate PRs to facilitate this kind of conversation.

Do we want to group these libraries in the AWS SDK (also adding to groups.aws-sdks.patterns), or is it better to keep them separate (by only including them in the allows list)?

We grouped all the AWS SDK in a group with the github.com/aws/aws-sdk-go-v2/* pattern to avoid dependencies problems like #39454 (a PR accidentally upgraded only the core library, breaking main).

[zmoog]

I suggest adding github.com/aws/aws-lambda-go and github.com/aws/smithy-go as dependencies to check if we are unsure. We can still add them later to the group.

[rowlandgeoff]

I rebased on main to get the changes we made for Azure and GCP.

@rowlandgeoff, would you mind taking a final look at the AWS config?

Happy to help if you need to be unblocked, but will defer to engineers closer to the code. Based on the comments, looks like sign-off will wait for now.

[zmoog]

Happy to help if you need to be unblocked, but will defer to engineers closer to the code. Based on the comments, looks like sign-off will wait for now.

Yeah, @rowlandgeoff sorry for the ping. The conversation got a new (and welcome) life right after I mentioned you. I'll request a new review when we complete the changes.

[zmoog]

@rowlandgeoff, I guess we're now ready for your review. We can make additional changes in future PRs as needed.

[rowlandgeoff]

LGTM, but I will defer to @dliappis since he's back from PTO

[zmoog]

Thanks! @dliappis, waiting for your feedback before merging the PR, then! 😇

[dliappis]

LGTM still from an eng prod PoV