-
Notifications
You must be signed in to change notification settings - Fork 893
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Versioned does not work for spark.SparkDataSet #1801
Comments
I'm also having this issue, in my case when saving to S3. I think it's due to the way the SparkDataSet sets its |
Thanks for reporting this! We'll take this into our sprint work, but we'd also be happy to accept a PR for this 🙂 |
Hi @Spectren, I've tried this out and versioned As for with s3, @alamastor, versioned Closing this issue but feel free to re-open if this is not resolved. :) |
@jmholzer confirmed still an issue on Azure databricks |
Closing this in favor of kedro-org/kedro-plugins#117, #2323 and kedro-org/kedro-plugins#114 I am quite confident this should work now, we've added warning and improve the documentation for using it correctly with Databricks. Since this issues mixed with many different issue (i.e. permission issue with S3, incorrect path on dbfs etc) , if there are problem with this, feel free to open a new issue |
Description
Versioning does not work for spark.SparkDataSet. It will save the version, but immediately after saving it will give the error that it does not exist (although it does and can be read by hand). I'm a newbie, so I might be doing something wrong, however, according to the documentation, everything should be correct.
Context
I wanted to save the processed dataset with the new version
Steps to Reproduce
Expected Result
The code will continue to work after saving the dataset version
Actual Result
VersionNotFoundError: Did not find any versions for
SparkDataSet(file_format=parquet,
filepath=/data/inc/.../result, load_args={},
save_args={'mode': overwrite}, version=Version(load=None,
save='2022-08-22T18.30.55.332Z'))
Your Environment
The text was updated successfully, but these errors were encountered: