Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(deps): update rust crate deltalake to 0.21 #129

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(deps): update rust crate deltalake to 0.21 #129

wants to merge 1 commit into from

Conversation

renovate[bot]
Copy link
Contributor

@renovate renovate bot commented Jun 14, 2024 •
edited
Loading

This PR contains the following updates:

Package Type Update Change
deltalake dependencies minor 0.17 -> 0.21

Release Notes

delta-io/delta.rs (deltalake)

v0.20.1

Full Changelog

Implemented enhancements:

  • Allow to specify Azurite hostname and service port as backend #​2900
  • docs section usage/Managing a table is out of date w.r.t. optimizing tables #​2891
  • generate more sensible row group size #​2545

Fixed bugs:

  • Cannot write to Minio with deltalake.write_deltalake or Polars #​2894
  • Schema Mismatch Error When appending Parquet Files with Metadata using Rust Engine #​2888
  • Assume role support has been broken since 2022 🤣 #​2879
  • z-order fails on table that is partitioned by value with space #​2834
  • "builder error for url" when creating an instance of a DeltaTable which is located in an azurite blob storage #​2815

Closed issues:

  • delta-rs can't write to a table if datafusion is not enabled #​2910

v0.20.0

Full Changelog

Fixed bugs:

  • DeltaTableBuilder flags ignored #​2808
  • Require files in config is not anymore used to skip reading add actions #​2796

Merged pull requests:

v0.19.1

Full Changelog

Implemented enhancements:

  • question: deletionVectors support #​2829
  • [Minor] Make Add::get_json_stats public #​2821
  • expose target_file_size in python side for WriterProperties #​2810
  • expose default_column_properties, column_properties of parquet WriterProperties in python #​2785
  • CDC support in deltalog when writing delta table #​2720
  • Function behaving similarly to SHOW PARTITIONS in the Python API #​2671
  • Expose set_statistics_truncate_length via Python WriterProperties #​2630

Fixed bugs:

  • write_deltalake with predicate throw index out of bounds #​2867
  • writing to blobfuse has stopped working in 0.19.2 #​2860
  • cannot read from public GCS bucket if non logged in #​2859
  • Stats missing for dataSkippingStatsColumns when escaping column name #​2849
  • 0.19.2 install error when using poetry, pdm on Ubuntu #​2848
  • deltalake-* crates use different version than specified in Cargo.toml, leading to unexpected behavior #​2847
  • Databricks fails integrity check after compacting with delta-rs #​2839
  • "failed to load region from IMDS" back in 0.19 despite AWS_EC2_METADATA_DISABLED=true #​2819
  • min/max_row_groups not respected #​2814
  • Large Memory Spike on Merge #​2802
  • Deleting large number of records fails with no error message #​2798
  • max_spill_size incorrect default value #​2794
  • Delta-RS Saved Delta Table not properly ingested into Databricks #​2779
  • Missing Linux binary releases and source tarball for Python release v0.19.0 #​2777
  • Transaction log parsing performance regression #​2760
  • RecordBatchWriter only creates stats for the first 32 columns; this prevents calling create_checkpoint. #​2745
  • DeltaScanBuilder does not respect datafusion context's datafusion.execution.parquet.pushdown_filters #​2739
  • IN (...) clauses appear to be ignored in merge commands with S3 - extra partitions scanned #​2726
  • Trailing slash on AWS_ENDPOINT raises S3 Error #​2656
  • AsyncChunkReader::get_bytes error: Generic MicrosoftAzure error: error decoding response body #​2592

v0.19.0

Full Changelog

Implemented enhancements:

Fixed bugs:

  • Write also insert change types in writer CDC #​2750
  • Regression in Python multiprocessing support #​2744
  • SchemaError occurs during table optimisation after upgrade to v0.18.1 #​2731
  • AWS WebIdentityToken exposure in log files #​2719
  • Write performance degrades with multiple writers #​2683
  • Write monotonic sequence, but read is non monotonic #​2659
  • Python write_deltalake with schema_mode="merge" casts types #​2642
  • Newest docs (potentially) not released #​2587
  • CDC is not generated for Structs and Lists #​2568

Closed issues:

Merged pull requests:

v0.18.2

Full Changelog

Implemented enhancements:

  • Choose which columns to store min/max values for #​2709
  • Projection pushdown for load_cdf #​2681
  • Way to check if Delta table exists at specified path #​2662
  • Support HDFS via hdfs-native package #​2611
  • Deletion _change_type does not appear in change data feed #​2579

Fixed bugs:

  • Slow add_actions.to_pydict for tables with large number of columns, impacting read performance #​2733
  • append is deleting records #​2716
  • segmentation fault - Python 3.10 on Mac M3 #​2706
  • Failure to delete dir and files #​2703
  • DeltaTable.from_data_catalog not working #​2699
  • Project should use the same version of ruff in the lint stage of python_build.yml as in pyproject.toml #​2678
  • un-tracked columns are giving json error when pyarrow schema have feild with nullable=False and create_checkpoint is trigged #​2675
  • [BUG]write_delta({'custom_metadata':str}) cannot be converted. str to pyDict error (0.18.2_DeltaPython/Windows10) #​2697
  • Pyarrow engine not supporting schema overwrite with Append mode #​2654
  • deltalake-core version re-exported by deltalake different than versions used by deltalake-azure and deltalake-gcp #​2647
  • i32 limit in JSON stats #​2646
  • Rust writer not encoding correct URL for partitions in delta table #​2634
  • Large Types breaks merge predicate pruning #​2632
  • Getting error when converting a partitioned parquet table to delta table #​2626
  • Arrow: Parquet does not support writing empty structs when creating checkpoint #​2622
  • InvalidTableLocation("Unknown scheme: gs") on 0.18.0 #​2610
  • Unable to read delta table created using Uniform #​2578
  • schema merging doesn't work when overwriting with a predicate #​2567

Closed issues:

  • Unable to write new partitions with type timestamp on tables created with delta-rs 0.10.0 #​2631

Merged pull requests:

v0.18.0

Full Changelog

Implemented enhancements:

  • documentation: concurrent writes for non-S3 backends #​2556
  • pyarrow options for write_delta #​2515
  • [deltalake_aws] Allow configuring separate endpoints for S3 and DynamoDB clients. #​2498
  • Include file stats when converting a parquet directory to a Delta table #​2490
  • Adopt the delta kernel types #​2489

Fixed bugs:

  • raise_if_not_exists for properties not configurable on CreateBuilder #​2564
  • write_deltalake with rust engine fails when mode is append and overwrite schema is enabled #​2553
  • Running the basic_operations examples fails with Error: Transaction { source: WriterFeaturesRequired(TimestampWithoutTimezone) } #​2552
  • invalid peer certificate: BadSignature when connecting to s3 from arm64/aarch64 #​2551
  • load_cdf() issue : Generic S3 error: request or response body error: operation timed out #​2549
  • write_deltalake fails on Databricks volume #​2540
  • Getting "Microsoft Azure Error: Operation timed out" when trying to retrieve big files #​2537
  • Impossible to append to a DeltaTable with float data type on RHEL #​2520
  • Creating DeltaTable object slow #​2518
  • write_deltalake throws parser error when using rust engine and big decimals #​2510
  • TypeError: Object of type int64 is not JSON serializable when writing using a Pandas dataframe #​2501
  • unable to read delta table when table contains both null and non-null add stats #​2477
  • Commits on WriteMode::MergeSchema cause table metadata corruption #​2468
  • S3 object store always returns IMDS warnings #​2460
  • File skipping according to documentation #​2427
  • LockClientError #​2379
  • get_app_transaction_version() returns wrong result #​2340
  • Property setting in create is not handled correctly #​2247
  • Handling of decimals in scientific notation #​2221
  • Unable to append to delta table without datafusion feature #​2204
  • Decimal Column with Value 0 Causes Failure in Python Binding #​2193

Merged pull requests:

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

  • If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

@renovate renovate bot changed the title fix(deps): update rust crate deltalake to 0.18 fix(deps): update rust crate deltalake to 0.19 Aug 15, 2024
@renovate renovate bot changed the title fix(deps): update rust crate deltalake to 0.19 fix(deps): update rust crate deltalake to 0.20 Sep 18, 2024
@renovate renovate bot changed the title fix(deps): update rust crate deltalake to 0.20 fix(deps): update rust crate deltalake to 0.21 Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
0 participants