Strategy to add new table base on silver data

Joe1912 — Thu, 02 Nov 2023 10:42:17 GMT

I have a merge function for streaming foreachBatch kind of
mergedf(df,i):

merge_func_1(df,i)

merge_func_2(df,i)

Then I want to add new merge_func_3 into it.

Is there any best practices for this case? when streaming always runs, how can I process data from beginning for merge_func_3 without stopping streaming then create another temp job to run for func_3, then run streaming again with adding func_3

topic Strategy to add new table base on silver data in Data Engineering

Strategy to add new table base on silver data