Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON analyzer assigns TIMESTAMP incorrectly for numeric data #1197

Closed
pmm-motif opened this issue Mar 19, 2023 · 2 comments
Closed

JSON analyzer assigns TIMESTAMP incorrectly for numeric data #1197

pmm-motif opened this issue Mar 19, 2023 · 2 comments

Comments

@pmm-motif
Copy link

I hit a weird error when using DuckDB WASM via insertJSONFromPath with JSON files. TLDR, one of the fields contained a timestamp in milliseconds which was marked as Timestamp[s] by DuckDB WASM and returned in microseconds. I found it a bit unexpected (with my data) that:

  1. DuckDB itself recognized the timestamp (rather than just return the int or float)
  2. The unit was incorrect

It was even more unexpected to notice that this behavior was rather inconsistent and was depending on the number of records included in JSON file. Sometimes it just returned INTEGER but in other times it returned TIMESTAMP. After some further digging, I was able to create a minimal file, which shows the unexpected behavior: https://gist.github.com/pmm-motif/762bf9878a2f80629e664ba9534896d9

After loading the JSON via insertJSONFromPath(...) and doing DESCRIBE TABLE <table-name> the type for ts is TIMESTAMP_S (rather than, say, INTEGER, which I was anticipating).

I suspect the reason might be related to scoring algorithm. I think it's actually wrong to try parsing any numeric data as timestamps, since:

My proposal is to actually do not try auto-parsing number as timestamp

@ankoh
Copy link
Member

ankoh commented Mar 20, 2023

Hotfixed in 653d226 . Please reopen if the issue persists.

@ankoh ankoh closed this as completed Mar 20, 2023
@pmm-motif
Copy link
Author

Whoa! That was quick! Many thanks @ankoh !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants