How to set up partitions on the streaming Delta Table?

[Leszek]

Let's assume that we have 3 streaming Delta Tables:

• Bronze
• Silver
• Gold

My aim is to add partitioning to Silver table (for example by Date).

So, as a result Gold table with throw an error that source table has been updated and I would need to set 'ignoreChanges' to 'true' option in the streaming. Then streaming will work but it will move all data from Silver (all files has been changed) to Gold resulting in duplicates.

My question is how to handle this problem in the best way?

Is it possible to manipulate streaming checkpoint somehow?

[-werners-]

is the change data feed functionality (of your silver table) an option, combined with merge in your gold table?

https://docs.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed

[Leszek]

I never used CDC in Databricks but my ELT architecture is based on streaming and I don't want to change that.

[Kaniz]

Hi @Leszek​ , We haven’t heard from you on the last response from @Werner Stinckens​ , and I was checking back to see if his suggestions helped you.

Or else, If you have any solution, please do share that with the community as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.