Error while encoding: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayDa

Get Started Discussions

Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.

Hello:)

we are trying to run an existing working flow that works currently on EMR, on databricks.

we use LTS 10.4, and when loading the data we get the following error:

at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:365)
Caused by: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayData is not a valid external type for schema of array<struct<id:string,name:string,name_type:string>>
at com.databricks.sql.expressions.codegen.RuntimeUtils.assertTrue(RuntimeUtils.java:77)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.MapObjects_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:233)
... 19 more

it's important to note that the same code works on emr with spark version 3.2.1. we tried looking at the data, removing some of the columns - but the data is consistent with the structure.

did anyone encounter this and could provide assistance in identifying the cause? we'd be happy to share more of the stack if relevant.

Thanks in advance!