Hello @pepco !

I will share with you my personal experience about a very similar behaviour I got like you.

If you check DBKS doc you will find that  git_source and task source: GIT are not recommended for DAB because local relative paths may not point to the same content in the git repo and bundles expect the deployed job to run from the same files that were deployed from the local bundle copy. 

You need to use workspace source for bundle tasks instead. https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-types

In my case, this is what I had :

mono repo
DAB deployment
source = Git branch/tag
custom local modules
no wheels
imports like: import lib

I understood at that tome that combination can work most of the time but it depends heavily on what DBKS puts into cwd or  sys.path for that specific task run.

So I took time to understand what is happening behind and in reality when a task uses git source, DBKS retrieves the notebook or python file from the git repo at runtime. For python script tasks, DBKS says that workspace paths must be absolute, while git paths are relative so if source is empty the task uses GIT when git_source is defined. 

My task file was found but my shared lib folder is not consistently on sys.path, so python was failing with the famous error no module named lib ....

A retry can succeed because the run may start in a slightly different initialized state or because the cluster already has a path state from a previous successful run. That does not mean the setup is deterministic.

What I have done so far is I started avoiding runtime git source for python imports and used source: WORKSPACE

(you can also remove git_source and let the bundle deploy the code into the workspace that works fine)

If you have a mono repo, you can use sync.paths so the shared code is deployed together with the bundle.