Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compat: Ensure that pandas dtype matches dask when loading data from parquet #156

Merged
merged 10 commits into from
Nov 12, 2024
Merged

compat: Ensure that pandas dtype matches dask when loading data from parquet #156

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
da02a4d
Ensure that dask dtype matches pandas in read_parquet
hoxbro Aug 17, 2024
955a607
Do conversion lower down
hoxbro Aug 17, 2024
7484028
Use dask util to determine conversion
hoxbro Aug 17, 2024
1a05f8a
Add simple test to check pyarrow_strings_enabled exists
hoxbro Aug 18, 2024
4962bc8
Add self_destruct True
hoxbro Aug 18, 2024
7758d9a
Try to add test
hoxbro Aug 18, 2024
feb6fe2
Updater test
hoxbro Aug 18, 2024
a88b272
Also account for save as convert string
hoxbro Aug 19, 2024
aec415c
Merge branch 'main' into force_dask_dtype_parquet
hoxbro Aug 19, 2024
6c0f0e1
Merge branch 'main' into force_dask_dtype_parquet
hoxbro Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion spatialpandas/io/parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -457,7 +457,7 @@ def _perform_read_parquet_dask(
if delayed_partitions:
result = from_delayed(
delayed_partitions, divisions=divisions, meta=meta, verify_meta=False
)
).astype(meta.dtypes)
hoxbro marked this conversation as resolved.
Show resolved Hide resolved
else:
# Single partition empty result
result = from_pandas(meta, npartitions=1)
Expand Down