[BUG] Event parsing error: String length (...) exceeds the maximum length (20000000)

[tgravescs]

Describe the bug

Looks like Spark 3.5.0 changed the jackson version and now there is a max json string length configuration. One of our customers hit this issue running the qualification tool, see:

https://issues.apache.org/jira/browse/SPARK-47150

Note that if the event isn't critical it will just ignore it and go on, I assume if it is critical that app may not be qualified properly.
A temporary solution is to go back to 3.4.X. But we may want to look at setting that spark config to up the maximum allowed.

full stack trace:

24/08/24 07:33:26 ERROR EventUtils: Unknown exception while parsing an event

        | com.fasterxml.jackson.core.exc.StreamConstraintsException: String length (20030999) exceeds the maximum length (20000000)

        |       at com.fasterxml.jackson.core.StreamReadConstraints.validateStringLength(StreamReadConstraints.java:324)

        |       at com.fasterxml.jackson.core.util.ReadConstrainedTextBuffer.validateStringLength(ReadConstrainedTextBuffer.java:27)

        |       at com.fasterxml.jackson.core.util.TextBuffer.finishCurrentSegment(TextBuffer.java:939)

        |       at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:2240)

        |       at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:2206)

        |       at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:323)

        |       at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:572)

        |       at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:100)

        |       at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:25)

        |       at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)

        |       at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4867)

        |       at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3219)

        |       at org.apache.spark.util.JsonProtocol$.sparkEventFromJson(JsonProtocol.scala:850)

        |       at org.apache.spark.util.JsonProtocol.sparkEventFromJson(JsonProtocol.scala)

        |       at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

[amahussein]

I cannot find a way to set JSONOptions(maxStringLen) in spark configurations. This is a class parameter that needs to be set each time an new instance is created.
The only option we have is to use older spark release by default to avoid this corner case for other customers.
It will be nice if we can make that configurable because currently it is hardcoded.

[tgravescs]

so I'm not atually sure if that spark config is being used at all, we are using the JsonProtocol to read the json events. that has its own json mapper:

  protected val mapper: ObjectMapper = new ObjectMapper().registerModule(DefaultScalaModule)
    .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)

Which pehaps has a default the same max string len? Anyway, I do wonder if we could override that by making our own class

[amahussein]

After discussing this issue offline with @tgravescs , we decided to do the following:

• Set the default spark version to older release for OnPrem. This requires:
  • test that no runtime error can be caused by using an older version of runtime with a version compiled with spark 3.5
• Allow the spark version to be overridden. i.e., add a new tools env_var that can override the default spark version
• Update the documentation to highlight that new config