madams
Contributor III

Thanks for providing that whole example, it was really easy to fiddle with.  I think I've found your solution.  Update the original two tasks on the job (if you want to keep them) like this:

      tasks:
        - task_key: notebook_task
          job_cluster_key: job_cluster
          notebook_task:
            notebook_path: ${workspace.file_path}/src/notebook.ipynb
            source: WORKSPACE
        
        - task_key: main_task
          depends_on:
            - task_key: notebook_task
          
          job_cluster_key: job_cluster
          python_wheel_task:
            package_name: my_project
            entry_point: main
          libraries:
            # By default we just include the .whl file generated for the my_project package.
            # See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
            # for more information on how to add other libraries.
            - whl: ${workspace.file_path}/dist/*.whl

Instead of using the `../` in the path, I used the variable `${workspace.file_path}` which references the deployed path.  I also added `source: WORKSPACE` to your notebook task so that it didn't default to GIT.

View solution in original post