Databricks Community

Zbyszek · yesterday

Hi
I'm trying to run my existing code which has worked on the older DB version.

CREATE TABLE IF NOT EXISTS catalog.demo.ABTHudi USING org.apache.hudi.Spark3DefaultSource OPTIONS ('primaryKey' = 'ID','hoodie.table.name' = 'ABTHudi')

AS SELECT * FROM parquet.`/mnt/bigdata/databricks/ABTParquet`;

Unfortunately on Maven there is no package for Spark 4.0 therefore I tried to use the "highest" available package, which is: org.apache.hudi:hudi-spark3.5-bundle_2.13:1.0.2

Spark configuration contains:

spark.serializer org.apache.spark.serializer.KryoSerializer
spark.sql.catalog.spark_catalog org.apache.spark.sql.hudi.catalog.HoodieCatalog
spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension

When the code above is executed it generates the following error:

Py4JJavaError: An error occurred while calling o686.save.
: org.apache.hudi.internal.schema.HoodieSchemaException: Failed to convert struct type to avro schema: StructType(StructField(BAD,IntegerType,true),StructField(ID,StringType,true),...,StructField(MORTDUE,DoubleType,true)

Conversion error is produced for every column from the parquet file. The same parquet file can be successfully written to Iceberg or Delta format.

Is there any chance to use Hudi format in DB 17 which is on Sark 4.0

I will appreciate any suggestions.

Thank You

Zbyszek