preAction in databricks while writing into Google Big Query Table?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-19-2023 08:58 PM
I am writing into Google Big Query table using append mode. I need to delete current day data before writing new data. I just want to know if there is any preActions parameter can be used to first delete data before writing into table? Below is the syntax but preActiones is not working in this:
(DF.write
.format("bigquery")
.option("temporaryGcsBucket", "Bucket-Name")
.option("preActions","Truncate table `Table-Name`")
.mode("append")
.option("credentialsFile",credentialfilepath)
.option("parentProject",projectName)
.option("table", "Table-Name")
.save()
)
- Labels:
-
Azure databricks
-
Google
-
GoogleBigQuery
-
Spark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-20-2023 07:32 AM
Can you use override mode instead append?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-20-2023 10:18 AM
Thanks @Camilla Lok for your valuable response. As I have mentioned in I just need to delete the current day data not the entire table data, I mentioned the Truncate query just for reference but we need to run delete query instead of Truncate.
(DF.write
.format("bigquery")
.option("temporaryGcsBucket", "Bucket-Name")
.option("preActions",'delete from Table_Name where Date_Col="2023-01-20"`')
.mode("append")
.option("credentialsFile",credentialfilepath)
.option("parentProject",projectName)
.option("table", "Table-Name")
.save()
)
But this delete query is not working in the above query.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2023 01:07 AM
Sorry, I didn't have to unread. Unfortunately I am not not sure if such option exists within the write command for data frame The only way out I can see is to delete the data before landin 😞

