[CHORE] Disables anonymous mode for S3 accesses in DeltaLake (#1975)

DeltaLake SDK does not support anonymous mode access (see: issue delta-io/delta-rs#1554) We throw an error if a user attempts to supply `anonymous=True`. --------- Co-authored-by: Jay Chia <[email protected]@users.noreply.github.com>
Eventual-Inc · Mar 5, 2024 · 9ed9b3f · 9ed9b3f
1 parent acd1fad
commit 9ed9b3f
Showing 1 changed file with 4 additions and 0 deletions.
diff --git a/daft/delta_lake/delta_lake_scan.py b/daft/delta_lake/delta_lake_scan.py
@@ -206,6 +206,10 @@ def _s3_config_to_storage_options(s3_config: S3Config) -> dict[str, str]:
         storage_options["allow_invalid_certificates"] = "false" if s3_config.verify_ssl else "true"
     if s3_config.connect_timeout_ms is not None:
         storage_options["connect_timeout"] = str(s3_config.connect_timeout_ms) + "ms"
+    if s3_config.anonymous:
+        raise ValueError(
+            "Reading from DeltaLake does not support anonymous mode! Please supply credentials via your S3Config."
+        )
     return storage_options