Create a Hudi table with Databrick 17

Zbyszek — Mon, 17 Nov 2025 16:07:36 GMT

Hi
I'm trying to run my existing code which has worked on the older DB version.

CREATE TABLE IF NOT EXISTS catalog.demo.ABTHudi USING org.apache.hudi.Spark3DefaultSource OPTIONS ('primaryKey' = 'ID','hoodie.table.name' = 'ABTHudi')

AS SELECT * FROM parquet.`/mnt/bigdata/databricks/ABTParquet`;

Unfortunately on Maven there is no package for Spark 4.0 therefore I tried to use the "highest" available package, which is: org.apache.hudi:hudi-spark3.5-bundle_2.13:1.0.2

Spark configuration contains:

spark.serializer org.apache.spark.serializer.KryoSerializer
spark.sql.catalog.spark_catalog org.apache.spark.sql.hudi.catalog.HoodieCatalog
spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension

When the code above is executed it generates the following error:

Py4JJavaError: An error occurred while calling o686.save.
: org.apache.hudi.internal.schema.HoodieSchemaException: Failed to convert struct type to avro schema: StructType(StructField(BAD,IntegerType,true),StructField(ID,StringType,true),...,StructField(MORTDUE,DoubleType,true)

Conversion error is produced for every column from the parquet file. The same parquet file can be successfully written to Iceberg or Delta format.

Is there any chance to use Hudi format in DB 17 which is on Sark 4.0

I will appreciate any suggestions.

Thank You

Zbyszek

Re: Create a Hudi table with Databrick 17

AbhaySingh — Tue, 18 Nov 2025 14:22:56 GMT

Support for Spark 4.0 is still an open issuse

https://issues.apache.org/jira/browse/HUDI-7915

Please use the preview for testing but you may have to think about design for production with supported releases.

Re: Create a Hudi table with Databrick 17

Zbyszek — Wed, 19 Nov 2025 11:58:57 GMT

Thank You for your response, I will wait for more updates on that.

Regards
Ziggy

topic Re: Create a Hudi table with Databrick 17 in Data Engineering

Create a Hudi table with Databrick 17

Re: Create a Hudi table with Databrick 17

Re: Create a Hudi table with Databrick 17