Error while encoding: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayDa

[liormayn]

Hello:)

we are trying to run an existing working flow that works currently on EMR, on databricks.

we use LTS 10.4, and when loading the data we get the following error:

at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:365)
Caused by: java.lang.RuntimeException: org.apache.spark.sql.catalyst.util.GenericArrayData is not a valid external type for schema of array<struct<id:string,name:string,name_type:string>>
at com.databricks.sql.expressions.codegen.RuntimeUtils.assertTrue(RuntimeUtils.java:77)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.MapObjects_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:233)
... 19 more

it's important to note that the same code works on emr with spark version 3.2.1. we tried looking at the data, removing some of the columns - but the data is consistent with the structure.

did anyone encounter this and could provide assistance in identifying the cause? we'd be happy to share more of the stack if relevant.

Thanks in advance!

[Kaniz]

Hi @liormayn, It seems you’re encountering an issue related to the schema of your data when running your existing workflow on Databricks.

Let’s explore some potential solutions:

1. Parquet Decimal Columns Issue:

   • The error message you’re seeing might be related to decimal-type columns in your source data. If your data contains decimal columns, there’s a known issue with the vectorized Parquet reader in Spark.
   • To address this, you can disable the vectorized Parquet reader by setting spark.sql.parquet.enableVe...¹.
   • This change should be made at the cluster level.

2. Column Order in Joins:

   • Sometimes, changing the order of joins can resolve issues related to schema compatibility. If you’re performing joins in your workflow, try reversing the order of the join operations.
   • Instead of df1.join(df2, on_col="A"), try df2.join(df1, on_col="A")².

3. Cluster Mode and Library Methods:

   • If you’re using a High Concurrency cluster with credential pass-through enabled, certain library methods may trigger errors. In such cases, consider using a different cluster mode³.

Feel free to share more details about your stack or any additional error messages, and I’ll be happy to assist further! 😊

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!