Hi Dear Team,
I am trying to import data from databricks to Exasol DB.
I am using following code in below with Spark version is 3.0.1 ,
dfw.write \
.format("jdbc") \
.option("driver", exa_driver) \
.option("url", exa_url) \
.option("dbtable", "table") \
.option("user", username) \
.option("password", exa_password) \
.option("truncate", "true") \
.option("numPartitions", "1") \
.option("fetchsize", "100000") \
.mode("overwrite") \
.save()
The problem is when mode is "overwrite", it always drop target table in Exasol db, although in the spark documentation (https://spark.apache.org/docs/3.0.1/sql-data-sources-jdbc.html#content) it says for "truncate" option that
truncate --> This is a JDBC writer related option. When SaveMode.Overwrite is enabled, this option causes Spark to truncate an existing table instead of dropping and recreating it. This can be more efficient, and prevents the table metadata (e.g., indices) from being removed. However, it will not work in some cases, such as when the new data has a different schema. It defaults to false. This option applies only to writing.
According to this explanation , I would expect with option("truncate", "true"), it should not drop but truncate the table. Nevertheless it drops the table even in that case. Note: we can have separate truncate command and go with append mode but I do not want to have extra second command but solve in one command as suggested in Exasol documentation here (https://github.com/exasol/spark-exasol-connector/blob/main/doc/user_guide/user_guide.md#spark-save-m...) as well.
Am I missing something or do you have any resolution ?
