-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update data directory structure #22
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few comments since this diff has some unrelated changes in it
) | ||
generate.fn.client.restart() | ||
generate.fn.client.restart(wait_for_workers=False) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an unrelated bugfix for running locally (xref dask/distributed#8534)
|
||
if str(path).startswith("s3://"): | ||
session = botocore.session.Session() | ||
creds = session.get_credentials() | ||
con.install_extension("httpfs") | ||
con.load_extension("httpfs") | ||
con.sql( | ||
f""" | ||
SET s3_region='{REGION}'; | ||
SET s3_access_key_id='{creds.access_key}'; | ||
SET s3_secret_access_key='{creds.secret_key}'; | ||
SET s3_session_token='{creds.token}'; | ||
""" | ||
) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is unrelated to the directory structure. We don't need to do this configuration because we're not reading / writing with duckdb.
# Whether to run data-processing tasks locally | ||
# or on the cloud with Coiled. | ||
local: true | ||
# Output location for data files. Can be a local directory | ||
# or a remote path like "s3://path/to/bucket". | ||
data-dir: ./data |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moving this configuration out into a standalone file (related to, but doesn't close, #2)
This PR simplifies the data directory structure a bit to something like this: