Hello Community,

I am currently working on populating gold layer tables. Source for these gold layer tables are silver layer tables. A query is going to run on silver layer tables, spark sql query contains joins between multiple tables.

ex:

select columns

from table1 

 inner join table2  

 on join_condition

 inner join table3 on join_condition

 where clause.

Now my question is how can i load the data incrementally from the query?. i should be able to schedule the pipeline for every 30 mins.

Thanks for the help.

Thanks

Venkat