- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-17-2021 04:56 AM
Currently using df.write.format("bigquery") ,Databricks only supports append and Overwrite modes in to writing Bigquery tables.Does Databricks has any option of executing the DMLs like Merge in to Bigquey using Databricks Notebooks.?
- Labels:
-
Bigquery
-
Databricks notebook
-
Delete
-
Merge
-
Update
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2021 09:40 AM
@Sumeet Dora , Unfortunately there is no direct "merge into" option for writing to Bigquery using Databricks notebook. You could write to an intermediate delta table using the "merge into" option in delta table. Then read from the delta table and perform a full overwrite to the bigquery table, so that you get the up-to-date latest info in delta table and as well in bigquery. Hope it helps
You can refer to "merge into" option of delta table here https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-merge-into.html#merge-into-...
Here are some example notebooks for writing to bigquery
https://docs.databricks.com/data/data-sources/google/bigquery.html#example-notebooks
We have filter push down, but those are used while reading the data from bigquery. In the meanwhile, if I find any alternatives, I will post here.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2021 09:40 AM
@Sumeet Dora , Unfortunately there is no direct "merge into" option for writing to Bigquery using Databricks notebook. You could write to an intermediate delta table using the "merge into" option in delta table. Then read from the delta table and perform a full overwrite to the bigquery table, so that you get the up-to-date latest info in delta table and as well in bigquery. Hope it helps
You can refer to "merge into" option of delta table here https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-merge-into.html#merge-into-...
Here are some example notebooks for writing to bigquery
https://docs.databricks.com/data/data-sources/google/bigquery.html#example-notebooks
We have filter push down, but those are used while reading the data from bigquery. In the meanwhile, if I find any alternatives, I will post here.
![](/skins/images/582998B45490C7019731A5B3A872C751/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/582998B45490C7019731A5B3A872C751/responsive_peak/images/icon_anonymous_message.png)