cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Backfill Delta table

Anonymous
Not applicable

What is the recommended way to backfill a delta table using a series of smaller date partitioned jobs?

1 ACCEPTED SOLUTION

Accepted Solutions

User16783855117
Contributor II
Another approach you might consider is creating a template notebook to query a known date range with widgets. For example, two date widgets, start time and end time. Then from there you could use Databricks Jobs to update these parameters for each run, and this way it will spin up a cluster for each date range, and you could run all of those clusters in parallel as well.

View solution in original post

1 REPLY 1

User16783855117
Contributor II
Another approach you might consider is creating a template notebook to query a known date range with widgets. For example, two date widgets, start time and end time. Then from there you could use Databricks Jobs to update these parameters for each run, and this way it will spin up a cluster for each date range, and you could run all of those clusters in parallel as well.