Allow regex in SOURCE_BUCKETS for environment variable #138
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This allow us to support regex in SOURCE_BUCKETS for the environment variable so that it matches a regex of bucket instead of CSV buckets. CSV has a limited string limit and doesn't allow user to enter a long list of buckets.
|
Hey @harriswong thanks for creating this pull request. I've added this for review into our next minor version release! |
|
This would be a huge improvement for me as well. I need to support a large number of buckets that could easily be defined via a REGEX. |
|
Thanks for your effort here, @harriswong You can refer to the recent changes here |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
#141
Description of changes:
This allow us to support regex in SOURCE_BUCKETS for the environment variable so that it matches a regex of bucket instead of CSV buckets.
Environment variable has limited string limit and doesn't allow us to enter a long list of buckets. (Imagine 50+ devs and each has their own bucket). Using regex won't have this character limit problem and is better for scaling.
I created a PR to demonstrate the idea of using regex in the field. It defaults to use
sourceBuckets[0]. Improvement and feedback are welcome.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.