Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Figure out the right long term way to deal with dates and timestamps in integration tests. #9747

Closed
revans2 opened this issue Nov 16, 2023 · 2 comments · Fixed by #9996
Assignees
Labels
test Only impacts tests

Comments

@revans2
Copy link
Collaborator

revans2 commented Nov 16, 2023

Is your feature request related to a problem? Please describe.
Python/Pyspark do not really support dates/timestamps that are prior to the year 1. This can cause some issues in some cases as Spark will happily produce values that cannot be shipped back to python. Even before #9736 the current code restricted the range of values that could be generated to try and avoid this issue. But there is nothing that prevents a test from subtracting a month, or year, or whatever from a timestamp that would result in a value being out of range.

I think it would be better to have timestamp and data gen produce a full range of possible supported values. Especially for corner case tests with timezones/etc. We could also have the code automatically convert dates and timestamps to the raw number representation using date_diff and unix_micros. But there was also the suggestion to try and do both. But I am not 100% sure how to do both of them and coordinate things. So this is here to try and figure out if it is possible or if there is a better solution we have not thought of yet.

@res-life
Copy link
Collaborator

res-life commented Dec 4, 2023

Should think the string gen together, refer to #9936

@sameerz
Copy link
Collaborator

sameerz commented Dec 19, 2023

As a follow on, revert forcing the DATAGEN_SEED to zero in issue #9701, issue #9820, issue #9916, issue #9936

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test Only impacts tests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants