Commits on Jan 26, 2024

1. PythonRunner Changes [databricks] (NVIDIA#10274) …

   * Download Maven from apache.org archives (NVIDIA#10225)
   
   Fixes NVIDIA#10224 
   
   Replace broken install using apt by downloading Maven from apache.org.
   
   Signed-off-by: Gera Shegalov <[email protected]>
   
   * Fix a hang for Pandas UDFs on DB 13.3[databricks] (NVIDIA#9833)
   
   fix NVIDIA#9493
   fix NVIDIA#9844
   
   The python runner uses two separate threads to write and read data with Python processes, 
   however on DB13.3, it becomes single-threaded, which means reading and writing run on the same thread.
   Now the first reading is always ahead of the first writing. But the original BatchQueue will wait
   on the first reading until the first writing is done. Then it will wait forever.
   
   Change made:
   
   - Update the BatchQueue to support asking for a batch instead of waiting unitl one is inserted into the queue. 
      This can eliminate the order requirement of reading and writing.
   - Introduce a new class named BatchProducer to work with the new BatchQueue to support rows number
      peek on demand for the reading.
   - Apply this new BatchQueue to relevant plans.
   - Update the Python runners to support writing one batch one time for the singled-threaded model.
   - Found an issue about PythonUDAF and RunningWindoFunctionExec, it may be a bug specific to DB 13.3,
      and add a test (test_window_aggregate_udf_on_cpu) for it.
   - Other small refactors
   ---------
   
   Signed-off-by: Firestarman <[email protected]>
   
   * Fix a potential data corruption for Pandas UDF (NVIDIA#9942)
   
   This PR moves the BatchQueue into the DataProducer to share the same lock as the output iterator
   returned by asIterator,  and make the batch movement from the input iterator to the batch queue be
   an atomic operation to eliminate the race when appending the batches to the queue.
   
   * Do some refactor for the Python UDF code to try to reduce duplicate code. (NVIDIA#9902)
   
   Signed-off-by: Firestarman <[email protected]>
   
   * Fixed 330db Shims to Adopt the PythonRunner Changes [databricks] (NVIDIA#10232)
   
   This PR removes the old 330db shims in favor of the new Shims, similar to the one in 341db. 
   
   **Tests:**
   Ran udf_test.py on Databricks 11.3 and they all passed. 
   
   fixes NVIDIA#10228 
   
   ---------
   
   Signed-off-by: raza jafri <[email protected]>
   
   ---------
   
   Signed-off-by: Gera Shegalov <[email protected]>
   Signed-off-by: Firestarman <[email protected]>
   Signed-off-by: raza jafri <[email protected]>
   Co-authored-by: Gera Shegalov <[email protected]>
   Co-authored-by: Liangcai Li <[email protected]>

   

   3 people authored Jan 26, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for dacc6fe
   • Browse repository at this point

   Copy the full SHA

   dacc6fe View commit details

   Browse the repository at this point in the history

2. Merge branch 'branch-23.12' into fix-merge …

   Signed-off-by: Jason Lowe <[email protected]>

   

   jlowe committed Jan 26, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 10ab52f
   • Browse repository at this point

   Copy the full SHA

   10ab52f View commit details

   Browse the repository at this point in the history