cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Apache Hudi Table creation using hudi maven library

ros
New Contributor III

I installed hudi maven library org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0

in Dbricks Runtime Ver : 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)

with spark config :

spark.sql.catalog.spark_catalog org.apache.spark.sql.hudi.catalog.HoodieCatalog
 
spark.serializer org.apache.spark.serializer.KryoSerializer
 
spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension

I ran this python cmd :

import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._

which gave me error :

ModuleNotFoundError: No module named 'org.apache.hudi'

And then i ran sql command in notebook :

create table hudi_cow_pt_tbl (
id bigint,
name string,
ts bigint,
dt string,
hh string
) using hudi
tblproperties (
type = 'cow',
primaryKey = 'id',
preCombineField = 'ts'
)
partitioned by (dt, hh)
location 's3://incred-databricks-data/hudi_dms_data/hudi_cow_pt_tbl';

which gives me error :

java.io.FileNotFoundException: No such file or directory: s3://incred-databricks-data/hudi_dms_data/hudi_cow_pt_tbl

where as this path exists : s3://incred-databricks-data/hudi_dms_data/

2 REPLIES 2

shan_chandra
Esteemed Contributor

@Roshan RC​ - can you please try with a mount location instead and let us know?

ros
New Contributor III

@Shanmugavel Chandrakasu​ 

%sql
create table hudi_cow_pt_tbl (
id bigint,
name string,
ts bigint,
dt string,
hh string
) using hudi
tblproperties (
type = 'cow',
primaryKey = 'id',
preCombineField = 'ts'
)
partitioned by (dt, hh)
location '/mnt/data/hudi_dms_data/hudi_cow_pt_tbl';

It still gave me error :

org.apache.hudi.exception.TableNotFoundException: Hoodie table not found in path Unable to find a hudi table for the user provided paths.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group