[FEA] JsonToStructs and ScanJson should invalidate the entire array on overflow

[revans2]

Is your feature request related to a problem? Please describe.

Right now when parsing an array of longs/ints/etc. If there is an overflow, we only invalidate the bad value, but Spark will invalidate the entire array.

 val df = Seq("""{"data": [9223372036854775807,-9223372036854775808]}}""","""{"data": [9223372036854775808, -9223372036854775809]}""","""{"data": [99999999999999999999999999999999999999, -99999999999999999999999999999999999999]}""", """{"data":[1,2,3,4,5,9223372036854775808]}""").toDF("json")

spark.conf.set("spark.rapids.sql.enabled", false)

df.repartition(1).selectExpr("from_json(json, 'data ARRAY<LONG>')").show(false)
+---------------------------------------------+
|from_json(json)                              |
+---------------------------------------------+
|{[9223372036854775807, -9223372036854775808]}|
|{null}                                       |
|{null}                                       |
|{null}                                       |
+---------------------------------------------+

spark.conf.set("spark.rapids.sql.enabled", true)
spark.conf.set("spark.rapids.sql.expression.JsonToStructs", true)

df.repartition(1).selectExpr("from_json(json, 'data ARRAY<LONG>')").show(false)
+---------------------------------------------+
|from_json(json)                              |
+---------------------------------------------+
|{[9223372036854775807, -9223372036854775808]}|
|{[null, null]}                               |
|{[null, null]}                               |
|{[1, 2, 3, 4, 5, null]}                      |
+---------------------------------------------+

We should try to match what Spark does.