Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip fastparquet timestamp tests when plugin cannot read/write timestamps #9831

Merged
merged 3 commits into from
Dec 1, 2023

Commits on Nov 22, 2023

  1. Skip fastparquet timestamp tests for non-UTC timezones.

    Fixes NVIDIA#9776.
    
    The tests in `fastparquet_compatibility_test.py` check for compatibility between
    Apache Spark, the Spark RAPIDS plugin, and fastparquet. In particular:
    1. `test_reading_file_written_by_spark_cpu` checks if timestamp columns written
        with Apache Spark are read similarly with fastparquet and the plugin.
    2. `test_reading_file_written_with_gpu` checks if timestamps written with
       the plugin are read the same on Apache Spark and fastparquet.
    
    If the timezone is not set to "UTC", and the system timezone isn't "UTC" either,
    the plugin falls back to CPU for read/write of Parquet timestamp columns. This would
    cause the above tests not to run: the plugin can neither read nor write timestamps
    on GPU.
    
    Further, fastparquet seems to interpret timestamps written from Spark as being
    in "UTC", regardless of the timezone settings. So on non-UTC timezones,
    Apache Spark and fastparquet get different results for the same input.
    
    For the two reasons above, it is best to only run the three-way timestamp comparison
    tests in setups with "UTC" timezone.
    
    This commit skips the timestamp tests described above, when a non-UTC timezone is
    detected.
    
    Signed-off-by: MithunR <[email protected]>
    mythrocks committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    8693bdb View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2023

  1. Configuration menu
    Copy the full SHA
    3d3c06b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7db470f View commit details
    Browse the repository at this point in the history