read_deltalake attempts to use S3 credentials for local files

[apostolos-geyer]

Describe the bug
A clear and concise description of what the bug is.

When attempting to read a local deltalake, daft will log multiple errors and attempt to retrieve S3 credentials, create a client for us-east-1, etc. Not sure if this is a bug or if there is some behaviour or configuration for daft to understand I'm working with local files and not to try to use S3, but I couldn't find anything about this in the docs. The file is still read successfully, but it would be nice to not have to wait for it to fail to get a session token, and attempt to create an S3 client.

To Reproduce
Steps to reproduce the behavior:

import daft
daft.read_deltalake('path/to/a/local/file')

output:

failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
S3 Credentials not provided or found when making client for us-east-1! Reverting to Anonymous mode. the credential provider was not enabled

Expected behavior
A clear and concise description of what you expected to happen.

The local deltalake should be read without attempting to use S3 or any other network locations and without logging errors.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• Daft Version: 0.3.2

If you guys are looking for contributors, I'd be happy to try and fix this myself. Never contributed to anything before so not sure if there's any procedures but if I can I'll give it a shot.

[jaychia]

Thanks @apostolos-geyer .... Good catch!

This definitely seems like a bug. Would LOVE to take a contribution ❤️

Here are some quick tips:

• Getting set up with Daft locally for development: https://github.com/Eventual-Inc/Daft/blob/main/CONTRIBUTING.md
• Here is where the Delta reads are getting triggered: https://github.com/Eventual-Inc/Daft/blob/main/daft/io/_deltalake.py#L18
• And here is where the issue is: https://github.com/Eventual-Inc/Daft/blob/main/daft/delta_lake/delta_lake_scan.py#L33-L68

We probably will want to only do the detection of credentials (S3Config.from_env()) only if the Delta path provided is an S3 path.

[jaychia]

Feel free to shoot us any questions about contributing :)

[jaychia]

Hey @apostolos-geyer any progress on this? Otherwise we might have to find someone on our end to fix it :)

[apostolos-geyer]

Hi sorry, had just started a co-op and had some other work and this slipped my mind. I will try it this week. ty

[apostolos-geyer]

@jaychia nevermind man too much on my plate right now my apologies for the hold up.

ill be back tho

[apostolos-geyer]

didnt mean to close

[kevinzwang]

This might actually be fixed by #3025 since it no longer attempts to populate the storage config with AWS parameters when it is not an s3 path. I tried it and it no longer seems to raise the same error anymore. However, I am still seeing this, will investigate:

daft.exceptions.DaftCoreException: DaftError::External Internal IO Error when Opening: /0-67ef78a7-31e3-4fa2-b6f1-859174a60a06-0.parquet:
Details:
No such file or directory (os error 2)

[kevinzwang]

Ah that turned out to be an unrelated issue with the metadata we write out with DataFrame.write_deltalake. @apostolos-geyer could you give it a try either by building Daft or updating to v0.3.9 once it comes out?

[kevinzwang]

v0.3.9 is released with the fix so I am closing this issue. If it still does not work for you feel free to reopen