cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks Design Help

venkat-bodempud
New Contributor III

Hello Community,

I am currently working on populating gold layer tables. Source for these gold layer tables are silver layer tables. A query is going to run on silver layer tables, spark sql query contains joins between multiple tables.

ex:

select columns

from table1 

 inner join table2  

 on join_condition

 inner join table3 on join_condition

 where clause.

Now my question is how can i load the data incrementally from the query?. i should be able to schedule the pipeline for every 30 mins.

Thanks for the help.

Thanks

Venkat

1 ACCEPTED SOLUTION

Accepted Solutions

Ajay-Pandey
Esteemed Contributor III

Hi @venkat,

You can use merge or upsert operation in databricks for the incremental load.

Yes you can schedule the job to run every 30 min by using databricks workflow.

View solution in original post

4 REPLIES 4

Ajay-Pandey
Esteemed Contributor III

Hi @venkat,

You can use merge or upsert operation in databricks for the incremental load.

Yes you can schedule the job to run every 30 min by using databricks workflow.

Hi @Ajay Pandey​ ,

Thanks for your reply,

I will try and let you know.

Thanks

Venkat

Ajay-Pandey
Esteemed Contributor III

Sure

Anonymous
Not applicable

Hi @bodempudi venkat​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!