feat(storage): azdls #223
feat(storage): azdls #223
Conversation
| @@ -9,6 +10,14 @@ pub(crate) async fn write_metadata_file( | |||
| table_metadata: impl Serialize, | |||
There was a problem hiding this comment.
Hm, ich glaube hier sollten wir nochmal ran in einem nächsten PR.
Die metadata_location ist ja schon Teil der table_metadata. Die aktuelle Implementierung lässt Fehler zu.
Darüber hinaus werden wir in Zukunft auch properties brauchen. Ein Kollege schaut sich gerade #156 an (@gongrilla), wofür wir properties brauchen.
twuebi
force-pushed
the
tp/azdls-storage
branch
from
f685347
to
65fc96e
Compare
twuebi
force-pushed
the
tp/azdls-storage
branch
from
65fc96e
to
37b6092
Compare
twuebi
force-pushed
the
tp/azdls-storage
branch
from
6e8d896
to
c2c4b6b
Compare
twuebi
force-pushed
the
tp/azdls-storage
branch
from
315c559
to
19ad604
Compare
| table_location, | ||
| secret | ||
| .ok_or_else(|| { | ||
| CredentialsError::MissingCredential(self.storage_type()) |
There was a problem hiding this comment.
By doing this we rule out the possibility of a credential-free adls - completely public. I am not sure if this is relevant- would the storage driver support it?
There was a problem hiding this comment.
Reqsign blows up if there's no credential, even worse, it'll take whatever it can find in env if nothing is passed to it which may cause really weird behavior if some az creds end up mounted into a catalog container, something like a profile being created for a warehouse based on the catalog provider's az account instead of from some client's account.
Co-authored-by: Christian <[email protected]>
azure datalake storage support, currently relies on forks of iceberg-rust -> opendal -> reqsign
we're also pulling in a lot of azure sdk dependencies just for the sas token stuff + validating the storage profile, once FileIO is up to speed we may wanna reconsider this, reqsign also has some sas facilities.