- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
You can solve it using the conditional tasks in the jobs. You can use If/Else conditional tasks in the main job configuration to run full/incremental based on the parameter. It allows you to route to different pipeline tasks based on the parameter value - all within job configuration with no code needed.
You can follow below
Create main lake flow job with 3 tasks as below
- Create an If/Else Conditional task- check_refresh_mode - Conditional task that evaluates {{job.parameters.full_refresh}} == "true"
- Create a pipeline task run_full_refresh to run the lakeflow connect pipeline (Select the Trigger a full refresh on the pipeline) - Pipeline task runs when condition full_refresh is true with complete data reingest
3. Create a pipeline task run_incremental to run the lakeflow connect pipeline (Dont select the Trigger a full refresh on the pipeline) - Pipeline task runs when condition full_refresh is false with incremental data ingest
You can set a schedule for the main lakeflow job to run incremental ingest with default full_refresh as false. Run the lakeflow job manually with full_refresh as true for full reingest