Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python] Table.from_arrow can't import nan values into a non-null float column #41812

Open
lord opened this issue May 24, 2024 · 0 comments
Open

Comments

@lord
Copy link
Contributor

lord commented May 24, 2024

Describe the bug, including details regarding any error messages, version, and platform.

This small examples fails with ValueError: Field pyarrow.Field<a: double not null> was non-nullable but pandas column had 1 null values on 16.1.0.

import pandas as pd
import pyarrow as pa

df = pd.DataFrame({"a": [1.0, float("nan")]})
schema = pa.schema([pa.field('a', pa.float64(), nullable=False)])
pa.Table.from_pandas(df, schema=schema)

I guess this seems like a bug to me, but I'm no pandas expert. It does feel like this makes roundtripping a non-null float column through pandas impossible?

Component(s)

Python

@lord lord added the Type: bug label May 24, 2024
@amoeba amoeba changed the title Table.from_arrow can't import nan values into a non-null float column [Python] Table.from_arrow can't import nan values into a non-null float column May 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant