-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(aws-s3): server access logging support automatic date-based partitioning #28141
Comments
I'll take this. |
@mrgrain This depends on an update to
|
I'm guessing we'll need to wait for the next AWS Service Spec Update PR to come through. That happens every Monday. Alternatively you can update If that doesn't work either, than the CFN schema might not be released yet. |
S3 server access logging newly supports date-based partitioning. This feature allows key formats of log objects to be partitioned by date, which is useful when querying from Athena. In CloudFormation, `TargetObjectKeyFormat` was added. This object has the `PartitionedPrefix` and the `SimplePrefix` property, only one of which is allowed. The `SimplePrefix` is the key format that originally existed. The `PartitionedPrefix` is the new key format and allows the user to select either the event time or delivery time as the date source. ```yaml Resources: S3BucketServerAccessLogEnabledTest: Type: AWS::S3::Bucket Properties: BucketName: access-log-test LoggingConfiguration: DestinationBucketName: myDestinationBucket LogFilePrefix: 'hello' TargetObjectKeyFormat: # You can deploy in either of the following formats SimplePrefix: {} # 1 PartitionedPrefix: {} # 2 PartitionedPrefix: # 3 PartitionDateSource: EventTime # | DeliveryTime ``` whats-new https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-s3-server-access-logging-date-partitioning CloudFormation User Guides https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-loggingconfiguration.html#cfn-s3-bucket-loggingconfiguration-targetobjectkeyformat S3 User Guides https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html#server-access-logging-overview Closes #28141 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
|
S3 server access logging newly supports date-based partitioning. This feature allows key formats of log objects to be partitioned by date, which is useful when querying from Athena. In CloudFormation, `TargetObjectKeyFormat` was added. This object has the `PartitionedPrefix` and the `SimplePrefix` property, only one of which is allowed. The `SimplePrefix` is the key format that originally existed. The `PartitionedPrefix` is the new key format and allows the user to select either the event time or delivery time as the date source. ```yaml Resources: S3BucketServerAccessLogEnabledTest: Type: AWS::S3::Bucket Properties: BucketName: access-log-test LoggingConfiguration: DestinationBucketName: myDestinationBucket LogFilePrefix: 'hello' TargetObjectKeyFormat: # You can deploy in either of the following formats SimplePrefix: {} # 1 PartitionedPrefix: {} # 2 PartitionedPrefix: # 3 PartitionDateSource: EventTime # | DeliveryTime ``` whats-new https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-s3-server-access-logging-date-partitioning CloudFormation User Guides https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-loggingconfiguration.html#cfn-s3-bucket-loggingconfiguration-targetobjectkeyformat S3 User Guides https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html#server-access-logging-overview Closes aws#28141 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
S3 server access logging newly supports date-based partitioning. This feature allows key formats of log objects to be partitioned by date, which is useful when querying from Athena. In CloudFormation, `TargetObjectKeyFormat` was added. This object has the `PartitionedPrefix` and the `SimplePrefix` property, only one of which is allowed. The `SimplePrefix` is the key format that originally existed. The `PartitionedPrefix` is the new key format and allows the user to select either the event time or delivery time as the date source. ```yaml Resources: S3BucketServerAccessLogEnabledTest: Type: AWS::S3::Bucket Properties: BucketName: access-log-test LoggingConfiguration: DestinationBucketName: myDestinationBucket LogFilePrefix: 'hello' TargetObjectKeyFormat: # You can deploy in either of the following formats SimplePrefix: {} # 1 PartitionedPrefix: {} # 2 PartitionedPrefix: # 3 PartitionDateSource: EventTime # | DeliveryTime ``` whats-new https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-s3-server-access-logging-date-partitioning CloudFormation User Guides https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-loggingconfiguration.html#cfn-s3-bucket-loggingconfiguration-targetobjectkeyformat S3 User Guides https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html#server-access-logging-overview Closes #28141 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Describe the feature
S3 Bucket Logging Configuration should support automatic date-based partitioning for delivered logs.
Use Case
See: https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-s3-server-access-logging-date-partitioning/
Proposed Solution
Probably need to introduce a new property and/or enum class to allow setting the TargetObjectKeyFormat
Other Information
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-partitionedprefix.html
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html
Acknowledgements
CDK version used
any
Environment details (OS name and version, etc.)
any
The text was updated successfully, but these errors were encountered: