Error while encoding: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayDa

Community Platform Discussions

Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.

Hello:)

we are trying to run an existing working flow that works currently on EMR, on databricks.

we use LTS 10.4, and when loading the data we get the following error:

at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:365)
Caused by: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayData is not a valid external type for schema of array<struct<id:string,name:string,name_type:string>>
at com.databricks.sql.expressions.codegen.RuntimeUtils.assertTrue(RuntimeUtils.java:77)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.MapObjects_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:233)
... 19 more

it's important to note that the same code works on emr with spark version 3.2.1. we tried looking at the data, removing some of the columns - but the data is consistent with the structure.

did anyone encounter this and could provide assistance in identifying the cause? we'd be happy to share more of the stack if relevant.

Thanks in advance!