You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Currently toJson is disabled by default because
it is experimental and has some known incompatibilities with Spark, and can be enabled by setting spark.rapids.sql.expression.StructsToJson=true.
Known issues are:
* There can be rounding differences when formatting floating-point numbers as strings. For example, Spark may produce -4.1243574E26 but the GPU may produce -4.124357351E26.
* Not all JSON options are respected
The goal of this is to go through all of the JSONOptions Specifically the write options and fall back to the CPU for things that we do not support. For example dropFieldIfAllNull, writeNullIfWithDefaultValue, and pretty.
Next we need to fall back to the CPU if we see any float or double values. Please note that we want a config to be added in that would let the user control if float/double values are allowed or not. This should be separate from spark.rapids.sql.castFloatToString.enabled. Finally StructToJson should be enabled by default.
We should then update the compatibility docs.
At a minimum we should have some tests to verify that we fall back at the right time for what we have added in, but it would be nice to see if we have time to add some more tests to verify that we are doing the right thing in as many cases as possible. Things like '\r', '\n', '\t' characters in strings...
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Currently toJson is disabled by default because
https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/docs/compatibility.md#to_json-function
The goal of this is to go through all of the JSONOptions Specifically the write options and fall back to the CPU for things that we do not support. For example
dropFieldIfAllNull
,writeNullIfWithDefaultValue
, andpretty
.Next we need to fall back to the CPU if we see any float or double values. Please note that we want a config to be added in that would let the user control if float/double values are allowed or not. This should be separate from
spark.rapids.sql.castFloatToString.enabled
. FinallyStructToJson
should be enabled by default.We should then update the compatibility docs.
At a minimum we should have some tests to verify that we fall back at the right time for what we have added in, but it would be nice to see if we have time to add some more tests to verify that we are doing the right thing in as many cases as possible. Things like '\r', '\n', '\t' characters in strings...
The text was updated successfully, but these errors were encountered: