kenny_hero
New Contributor III

Just to share knowledge with other readers, I think the best option for now(until Databricks supports adding project root to PYTHONPATH) is to refactor the project to have custom modules in a separate folder, create a wheel when building DAB and declare as dependency of pipeline/job config.

I followed the instruction in this page, but struggled to make it work. It turned out dependency config is different if pipeline runs on serverless. Instead of using "libraries" as in the instruction , I have to use "environment".

resources:
  pipelines:
    <pipeline name>
      environment:
        dependencies:
          - <path to wheel file>

I wish the instruction included this example for serverless, or Databricks supports same configuration regardless of compute type.

View solution in original post