Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for S3 Storage Class Configuration in AWS S3 Exporter #35173

Open
pollosp opened this issue Sep 13, 2024 · 2 comments
Open

Add Support for S3 Storage Class Configuration in AWS S3 Exporter #35173

pollosp opened this issue Sep 13, 2024 · 2 comments
Labels
enhancement New feature or request exporter/awss3 help wanted Extra attention is needed

Comments

@pollosp
Copy link

pollosp commented Sep 13, 2024

Component(s)

exporter/awss3

Is your feature request related to a problem? Please describe.

Problem

At present, the AWS S3 exporter in the OpenTelemetry Collector does not provide a way to specify the S3 storage class during uploads. For cost optimization and to meet various data retention requirements, users often need to store data in specific S3 storage classes such as:

STANDARD_IA (Standard-Infrequent Access)
ONEZONE_IA (One Zone-Infrequent Access)
GLACIER, DEEP_ARCHIVE, etc.

Without this feature, users have to manually configure lifecycle rules on their buckets or use external processes to transition data between storage classes, which adds complexity API costs and overhead.

Proposed Solution:
Modify the existing AWS S3 exporter to include support for setting the S3 storage class when uploading objects to S3. The storage class would be configurable in the exporter’s configuration file, similar to how other S3 options (like bucket name and key) are handled.
Something like:

	if config.S3Uploader.StorageClass != "" {
		storageClass = config.S3Uploader.StorageClass
	}

	_, err = uploader.Upload(&s3manager.UploadInput{
		Bucket:          aws.String(config.S3Uploader.S3Bucket),
		Key:             aws.String(key),
		Body:            reader,
		ContentEncoding: &encoding,
		StorageClass:    aws.String(storageClass), // Set the storage class here
	})

Proposal Implementation Details:

New Configuration Option:
Add a new storage_class field to the aws_s3 exporter configuration. This field would allow users to specify the desired storage class for their S3 objects.
Example YAML config:

exporters:
  aws_s3:
    bucket: "my-bucket"
    storage_class: "STANDARD_IA"  # New field to specify storage class

Modify S3 Upload Logic:
Update the S3 upload logic to include the StorageClass parameter when calling the S3 API. If the storage_class is not specified, it should default to STANDARD.

Backward Compatibility:
The change is backward compatible, with the STANDARD storage class used as the default value if the storage_class field is not provided.

Describe the solution you'd like

  • The user can specify the desired S3 storage class via the exporter’s configuration.
  • Objects uploaded to S3 will be stored in the specified storage class.
  • If no storage class is specified, the objects will default to the STANDARD storage class.
  • Proper documentation is updated to reflect the new configuration option.

Describe alternatives you've considered

No response

Additional context

No response

@pollosp pollosp added enhancement New feature or request needs triage New item requiring triage labels Sep 13, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@atoulme
Copy link
Contributor

atoulme commented Oct 2, 2024

Sounds good. Feel free to provide a patch for this enhancement request!

@atoulme atoulme added help wanted Extra attention is needed and removed needs triage New item requiring triage labels Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request exporter/awss3 help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants