balajij8
Contributor III

You can solve it using the conditional tasks in the jobs. You can use If/Else conditional tasks in the main job configuration to run full/incremental based on the parameter. It allows you to route to different pipeline tasks based on the parameter value - all within job configuration with no code needed.

You can follow below

 1_gXVbGJUQlqIZF7t9eDo4Yw

 Create main lake flow job with 3 tasks as below

 

  1. Create an If/Else Conditional task- check_refresh_mode - Conditional task that evaluates {{job.parameters.full_refresh}} == "true"
  2. Create a pipeline task run_full_refresh to run the lakeflow connect pipeline (Select the Trigger a full refresh on the pipeline) - Pipeline task runs when condition full_refresh is true with complete data reingest1_Bj3O-cOyJ-6Ee_ZqhyB7AA

     3. Create a pipeline task run_incremental to run the lakeflow connect pipeline (Dont select the Trigger a full refresh on the pipeline) - Pipeline task runs when condition full_refresh is false with incremental data ingest1_KBfdpt66QVGeNbxN6COJjQ

    You can set a schedule for the main lakeflow job to run incremental ingest with default full_refresh as false. Run the lakeflow job manually with full_refresh as true for full reingest