juanlozadab
Contributor

I think there might be another thing worth checking before adding the extra repo pull task.
In the job YAML you posted, the Git source is using:
git_branch: main
but in the code you posted later you're explicitly pulling:
BRANCH = "develop"
So I'd first check whether experiment.py actually exists in the exact main commit being used by the job. Databricks takes a snapshot of the configured branch when the run starts, and you can see the commit SHA used by the run in the job details.
I'd probably add something simple before the import just to verify what the job can actually see:
import os
print(os.getcwd())
print(os.listdir(os.getcwd()))
If experiment.py is really next to INIT_PROCESSING in that exact commit, then I'd expect it to show up there, and that would narrow this down quite a bit.
Given that your manual repo update is targeting develop while the job is targeting main, I'd check that first before introducing another task or the Repos API.

jlb