Issue while writing data to unity catalog using JDBC

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

While writing the data to a pre-existing table in the unity catalog using JDBC. it just writes the Delta of the data.

Driver used:

com.databricks:databricks-jdbc:2.6.36

Lets say I have the table has rows:

+-+-+
|a|b| 
+-+-+
|1|2|
|3|4|

and I am appending the row using .union

+-+-+
|a|b|
+-+-+
|1|2|
+-+-+

I am using a `.union` here after reading the table with the row to append which results to this in my logs

+-+-+
|a|b| 
+-+-+
|1|2|
|3|4|
+-+-+

but after writing to table with mode ".overwrite" using jdbc

write_options = {"url": catalog_jdbc_url,
                         "driver": "com.databricks.client.jdbc.Driver"}
 (df.write.format("jdbc")
 .options(**write_options)
 .option("truncate", "true")
 .option("dbtable", table_name)
 .save(mode="overwrite"))

the table just shows in the unity catalog: