BigQuery - Delete or update from Databricks?

[Kayla]

I'm trying to sync a delta table in Databricks to a BigQuery table. For the most part, appending is sufficient, but occassionally we need to overwrite rows - which we've only been able to do by overwriting the entire table.

Is there any way to do updates/deletes or partial overwrites to BigQuery from within Databricks?

[Anonymous]

@Kayla Pomakoy​ :

I would like you to use DBSQL capability instead of BigQuery. Feel free to get in touch with the SA if you are an existing customer 🙂 We can tell you value of using DBSQL.

Asides, yes there is a BigQuery API to create a connection between your Databricks notebook and the BigQuery table. To tackle update or delete specific rows in a BigQuery table, you can use SQL statements such as UPDATE or DELETE with a WHERE clause that specifies the rows to modify. Finally, Read the updated BigQuery table back into Databricks as a DataFrame using the BigQuery Connector for Spark.

[Anonymous]

Hi @Kayla Pomakoy​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!