Issue while writing data to unity catalog using JDBC

DumbBeaver — Thu, 04 Apr 2024 21:28:54 GMT

While writing the data to a pre-existing table in the unity catalog using JDBC. it just writes the Delta of the data.

Driver used:

com.databricks:databricks-jdbc:2.6.36

Lets say I have the table has rows:

+-+-+ |a|b| +-+-+ |1|2| |3|4|

and I am appending the row using .union

+-+-+ |a|b| +-+-+ |1|2| +-+-+

I am using a `.union` here after reading the table with the row to append which results to this in my logs

+-+-+ |a|b| +-+-+ |1|2| |3|4| +-+-+

but after writing to table with mode ".overwrite" using jdbc

write_options = {"url": catalog_jdbc_url, "driver": "com.databricks.client.jdbc.Driver"} (df.write.format("jdbc") .options(**write_options) .option("truncate", "true") .option("dbtable", table_name) .save(mode="overwrite"))

the table just shows in the unity catalog:

+-+-+ |a|b| +-+-+ |1|2| +-+-+

Is there any reason for this or config issue?

topic Issue while writing data to unity catalog using JDBC in Data Engineering

Issue while writing data to unity catalog using JDBC