NPE on CreateJacksonParser and Databricks 14.3LTS with Spark StructuredStreaming

joss — Mon, 04 Mar 2024 09:40:42 GMT

hello,

I have a spark StructuredStreaming job : the source is a kafka topic in json.

it work find with databricks 14.2, but when a change to 14.3LTS, I have a NPE in CreateJacksonParser:

Caused by: NullPointerException: at org.apache.spark.sql.catalyst.json.CreateJacksonParser$.internalRow(CreateJacksonParser.scala:93)

Do you have any idea what this error is?

Regards

Re: NPE on CreateJacksonParser and Databricks 14.3LTS with Spark StructuredStreaming

joss — Tue, 05 Mar 2024 09:07:50 GMT

Hi ,

thank you for your quick reply,

i found the problem :

val newSchema = spark.read.json(df.select("data").as[String]).schema

if "data" have 1 value to null, in 14.2 it work, but with 14.3LTS this function return a NPE

I don't know if it is a bug