Write mode features in Bigquey using Databricks notebook.

Sumeet_Dora — Fri, 17 Sep 2021 11:56:14 GMT

Currently using df.write.format("bigquery") ,Databricks only supports append and Overwrite modes in to writing Bigquery tables.Does Databricks has any option of executing the DMLs like Merge in to Bigquey using Databricks Notebooks.?

Re: Write mode features in Bigquey using Databricks notebook.

mathan_pillai — Wed, 20 Oct 2021 16:40:08 GMT

@Sumeet Dora , Unfortunately there is no direct "merge into" option for writing to Bigquery using Databricks notebook. You could write to an intermediate delta table using the "merge into" option in delta table. Then read from the delta table and perform a full overwrite to the bigquery table, so that you get the up-to-date latest info in delta table and as well in bigquery. Hope it helps

You can refer to "merge into" option of delta table here https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-merge-into.html#merge-into-delta-lake-on-databricks

Here are some example notebooks for writing to bigquery

https://docs.databricks.com/data/data-sources/google/bigquery.html#example-notebooks

We have filter push down, but those are used while reading the data from bigquery. In the meanwhile, if I find any alternatives, I will post here.

topic Write mode features in Bigquey using Databricks notebook. in Data Engineering

Write mode features in Bigquey using Databricks notebook.

Re: Write mode features in Bigquey using Databricks notebook.