Create a Hudi table with Databrick 17

Zbyszek
New Contributor II

‎11-17-2025 08:07 AM

Hi 
I'm trying to run my existing code which has worked on the older DB version.

CREATE TABLE IF NOT EXISTS catalog.demo.ABTHudi USING org.apache.hudi.Spark3DefaultSource OPTIONS ('primaryKey' = 'ID','hoodie.table.name' = 'ABTHudi')
AS SELECT * FROM parquet.`/mnt/bigdata/databricks/ABTParquet`;
 
Unfortunately on Maven there is no package for Spark 4.0 therefore I tried to use the "highest" available package, which is: org.apache.hudi:hudi-spark3.5-bundle_2.13:1.0.2
Spark configuration contains:
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.sql.catalog.spark_catalog org.apache.spark.sql.hudi.catalog.HoodieCatalog
spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
 
When the code above is executed it generates the following error:
Py4JJavaError: An error occurred while calling o686.save.
: org.apache.hudi.internal.schema.HoodieSchemaException: Failed to convert struct type to avro schema: StructType(StructField(BAD,IntegerType,true),StructField(ID,StringType,true),...,StructField(MORTDUE,DoubleType,true)
Conversion error is produced for every column from the parquet file. The same parquet file can be successfully written to Iceberg or Delta format.
Is there any chance to use Hudi format in DB 17 which is on Sark 4.0 
I will appreciate any suggestions.
 
Thank You
Zbyszek
 
 

 

Labels:
0 Kudos

AbhaySingh
Databricks Employee
Databricks Employee

‎11-18-2025 06:22 AM

Support for Spark 4.0 is still an open issuse

https://issues.apache.org/jira/browse/HUDI-7915

Please use the preview for testing but you may have to think about design for production with supported releases. 

View solution in original post

0 Kudos

Zbyszek
New Contributor II

‎11-19-2025 03:58 AM

Thank You for your response, I will wait for more updates on that.

Regards
Ziggy