topic Re: How to set up partitions on the streaming Delta Table? in Data Engineering

How to set up partitions on the streaming Delta Table?

Leszek — Mon, 29 Aug 2022 08:34:17 GMT

Let's assume that we have 3 streaming Delta Tables:

Bronze
Silver
Gold

My aim is to add partitioning to Silver table (for example by Date).

So, as a result Gold table with throw an error that source table has been updated and I would need to set 'ignoreChanges' to 'true' option in the streaming. Then streaming will work but it will move all data from Silver (all files has been changed) to Gold resulting in duplicates.

My question is how to handle this problem in the best way?

Is it possible to manipulate streaming checkpoint somehow?

Re: How to set up partitions on the streaming Delta Table?

-werners- — Mon, 29 Aug 2022 08:39:25 GMT

is the change data feed functionality (of your silver table) an option, combined with merge in your gold table?

https://docs.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed

Re: How to set up partitions on the streaming Delta Table?

Leszek — Fri, 09 Sep 2022 13:50:00 GMT

I never used CDC in Databricks but my ELT architecture is based on streaming and I don't want to change that.