Unable to run python script from git repo in Databricks job

[Divya_Bhadauria]

I'm getting cannot read python file on running this job which is configured to run a python script from git repo.

Run result unavailable: run failed with error message

Cannot read the python file /Repos/.internal/7c39d645692_commits/ff669d089cd8f93e98ccb96c513d5e675c2423f1/<Relative file path in git>

Please check driver logs for more details.

My question is why is it looking for the file internally (/Repos/.internal/) ?

[Anonymous]

@Divya Bhadauria​ :

The error message suggests that Databricks is trying to read the Python file from a path that starts with /Repos/.internal/. This indicates that the Databricks job is not able to download the Python file from the git repo specified in the job configuration.

To troubleshoot this issue, you can try the following steps:

1. Check that the git repo URL specified in the job configuration is correct and accessible. You can try to access the repo from your Databricks workspace to verify that it is accessible.
2. Check the branch, commit, or tag specified in the job configuration. Make sure that it corresponds to a valid version of the code that contains the Python file you are trying to execute.
3. Check the relative file path of the Python file specified in the job configuration. Make sure that it is correct and matches the actual file path in the git repo.
4. Check that the Databricks cluster has access to the internet and is able to connect to the git repo. You can try running a test command to download a file from the repo using a command like curl or wget
5. Check the driver logs for more information about the error. The driver logs may contain more detailed error messages or stack traces that can help identify the issue.

Once you have identified the issue, you can update the job configuration or make changes to the Databricks environment to resolve the issue.

[Anonymous]

Hi @Divya Bhadauria​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

[Divya_Bhadauria]

Hi Vidula,

Yes, the above solution worked out for me. Tried debugging using all of the above steps and it turned out the path I was using in the job config was incorrect.

[SakthiGanesh]

Hi @Divya_Bhadauria, I'm facing the same internal commit issue from my end. I don't gave any internal path in the databricks workflow. I gave the source to azure DevOps services with branch name. But when I ran the workflow it gives the below error and the driver log doesn't show any error. Not sure from where it takes the path. Also, I don't have made any Repos config in Databricks. Any idea on this?

Cannot read the python file /Repos/.internal/277bc54e62_commits/c422b52459f3ca31841c384b04b0635ceb9567d7/file_name.py. Please check driver logs for more details.

[Divya_Bhadauria]

Hi @SakthiGanesh , For some weird reason Git repo option didn’t work for me the job so I changed the source to python wheel and it resolved the issue.

[SakthiGanesh]

Thanks @Divya_Bhadauria . For me the issue is with the extension of the notebook and it is solved now.

I gave the source as Repos, type as python script. In repos all python files are shows with ".py" extension. So I thought it refers the correct path. But when I cloned the Repos to my workspace it referred the file as Notebook and the extension ".py" was not available. This is what the issue from my end. So, I modified the name with .py and then it shows the script as file instead of notebook. This solves the issue.