Error while encoding: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayDa

liormayn — Thu, 18 Apr 2024 11:03:28 GMT

Hello:)

we are trying to run an existing working flow that works currently on EMR, on databricks.

we use LTS 10.4, and when loading the data we get the following error:

at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:365) Caused by: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayData is not a valid external type for schema of array<struct<id:string,name:string,name_type:string>> at com.databricks.sql.expressions.codegen.RuntimeUtils.assertTrue(RuntimeUtils.java:77) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.MapObjects_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:233) ... 19 more

it's important to note that the same code works on emr with spark version 3.2.1. we tried looking at the data, removing some of the columns - but the data is consistent with the structure.

did anyone encounter this and could provide assistance in identifying the cause? we'd be happy to share more of the stack if relevant.

Thanks in advance!

topic Error while encoding: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayDa in Get Started Discussions

Error while encoding: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayDa