Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support create object store source tables without depending on environment variables #5731

Closed
r4ntix opened this issue Mar 25, 2023 · 0 comments · Fixed by #5732
Closed

Support create object store source tables without depending on environment variables #5731

r4ntix opened this issue Mar 25, 2023 · 0 comments · Fixed by #5732
Labels
enhancement New feature or request

Comments

@r4ntix
Copy link
Contributor

r4ntix commented Mar 25, 2023

Is your feature request related to a problem or challenge?

Currently we already support object store source table registration through sql:

CREATE EXTERNAL TABLE test
STORED AS PARQUET
LOCATION 's3://bucket/path/file.parquet';

However, some authentication information required by the object store is obtained through environment variables:

AWS_ACCESS_KEY_ID -> access_key_id
AWS_SECRET_ACCESS_KEY -> secret_access_key
AWS_DEFAULT_REGION -> region
AWS_ENDPOINT -> endpoint
AWS_SESSION_TOKEN -> token

$ export AWS_DEFAULT_REGION=us-east-2
$ export AWS_SECRET_ACCESS_KEY=***************************
$ export AWS_ACCESS_KEY_ID=**************
$ datafusion-cli

This method is not flexible, when we need to access and analyze data files under different accounts. We need to modify the environment variables and go back to the cli.

Describe the solution you'd like

We should support obtaining necessary information when creating an object store table through sql.
Using the already supported OPTIONS parameter, we can do this:

CREATE EXTERNAL TABLE test
STORED AS PARQUET
OPTIONS(
    'access_key_id' '******',
    'access_key' '******',
    'region' 'us-east-2'
)
LOCATION 's3://bucket/path/file.parquet';

Also, when the user does not specify options, we can continue to use the default environment variables.

Describe alternatives you've considered

No response

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
1 participant