emma_s
Databricks Employee
Databricks Employee

Hi, if you've already created the pipeline you don't need to create it again via the DAB, just get the pipeline id from the UI and pass that into your job. Also your syntax for the task and the job should be more like this:

  jobs:
    my_pipeline_job:
      name: my-pipeline-job
      tasks:
        - task_key: my-pipeline-task
          pipeline_task:
            pipeline_id: [pass your the pipeline of existing pipeline here]