While writing the data to a pre-existing table in the unity catalog using JDBC. it just writes the Delta of the data.
Driver used:
com.databricks:databricks-jdbc:2.6.36
Lets say I have the table has rows:
+-+-+
|a|b|
+-+-+
|1|2|
|3|4|
and I am appending the row using .union
+-+-+
|a|b|
+-+-+
|1|2|
+-+-+
I am using a `.union` here after reading the table with the row to append which results to this in my logs
+-+-+
|a|b|
+-+-+
|1|2|
|3|4|
+-+-+
but after writing to table with mode ".overwrite" using jdbc
write_options = {"url": catalog_jdbc_url,
"driver": "com.databricks.client.jdbc.Driver"}
(df.write.format("jdbc")
.options(**write_options)
.option("truncate", "true")
.option("dbtable", table_name)
.save(mode="overwrite"))
the table just shows in the unity catalog:
+-+-+
|a|b|
+-+-+
|1|2|
+-+-+
Is there any reason for this or config issue?
