- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-21-2021 07:56 AM
Hi Dear Team,
I am trying to import data from databricks to Exasol DB.
I am using following code in below with Spark version is 3.0.1 ,
dfw.write \
.format("jdbc") \
.option("driver", exa_driver) \
.option("url", exa_url) \
.option("dbtable", "table") \
.option("user", username) \
.option("password", exa_password) \
.option("truncate", "true") \
.option("numPartitions", "1") \
.option("fetchsize", "100000") \
.mode("overwrite") \
.save()
The problem is when mode is "overwrite", it always drop target table in Exasol db, although in the spark documentation (https://spark.apache.org/docs/3.0.1/sql-data-sources-jdbc.html#content) it says for "truncate" option that
truncate --> This is a JDBC writer related option. When SaveMode.Overwrite is enabled, this option causes Spark to truncate an existing table instead of dropping and recreating it. This can be more efficient, and prevents the table metadata (e.g., indices) from being removed. However, it will not work in some cases, such as when the new data has a different schema. It defaults to false. This option applies only to writing.
According to this explanation , I would expect with option("truncate", "true"), it should not drop but truncate the table. Nevertheless it drops the table even in that case. Note: we can have separate truncate command and go with append mode but I do not want to have extra second command but solve in one command as suggested in Exasol documentation here (https://github.com/exasol/spark-exasol-connector/blob/main/doc/user_guide/user_guide.md#spark-save-m...) as well.
Am I missing something or do you have any resolution ?
- Labels:
-
Spark
-
Spark Version
-
Table
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-12-2021 04:35 PM
Hi @Akif Cakir ,
You are correct, this is the expected behavior when using JDBC connector. Docs here
Have you try to use the "exasol" connector instead of JDBC? do you also get this same behavior?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-12-2021 04:35 PM
Hi @Akif Cakir ,
You are correct, this is the expected behavior when using JDBC connector. Docs here
Have you try to use the "exasol" connector instead of JDBC? do you also get this same behavior?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-12-2022 09:06 AM
Facing the same problem I used the following:
sfOptions = {
"sfURL" : "<account>.snowflakecomputing.com",
"sfAccount" : "<account>",
"sfUser" : "<user>",
"sfPassword" : "***",
"sfDatabase" : "<database>",
"sfSchema" : "<schema>",
"sfWarehouse" : "<warehouse>",
"truncate_table" : "ON",
"usestagingtable" : "OFF",
}
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-29-2023 09:39 AM
@AkifCakir , Were you able to find a way to truncate without dropping the table using the .write function as I am facing the same issue as well.

