Use Python code from a remote Git repository

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

I'm trying to create a task where the source is a Python script located in remote GitLab repo. I'm following the instructions HERE and this is how I have the task set up:

However, no matter what path I specify all I get is the error below:

Cannot read the python file /Repos/.internal/4ac77871e5_commits/36c5bd9261b47a0bad5eaf99f32b9a2cd7032471/projects/membership-churn/mlops/workflow_notebooks_v1/02%20-%20Production%20Data%20ETL.py. Please check driver logs for more details

I've searched the Community and found THIS and tried to follow the advice mentioned there.

The Logs show the below AssertionError:

23/09/21 19:25:24 WARN JupyterDriverLocal: User code returned error with traceback: [0;31m---------------------------------------------------------------------------[0m
[0;31mAssertionError[0m                            Traceback (most recent call last)
File [0;32m~/.ipykernel/1132/command--1-426573936:2[0m
[1;32m      1[0m [38;5;28;01mfrom[39;00m [38;5;21;01mos[39;00m[38;5;21;01m.[39;00m[38;5;21;01mpath[39;00m [38;5;28;01mimport[39;00m exists
[0;32m----> 2[0m [38;5;28;01massert[39;00m(exists([38;5;124m"[39m[38;5;124m/Workspace/Repos/.internal/4ac77871e5_commits/36c5bd9261b47a0bad5eaf99f32b9a2cd7032471/projects/membership-churn/mlops/workflow_notebooks_v1/02[39m[38;5;124m%[39m[38;5;124m20-[39m[38;5;124m%[39m[38;5;124m20Production[39m[38;5;124m%[39m[38;5;124m20Data[39m[38;5;132;01m%20E[39;00m[38;5;124mTL.py[39m[38;5;124m"[39m))

I'm not quite sure why is it trying to invoke a Jupyter Driver since the files in my repo are all *.py

What am I doing wrong here?

1 REPLY 1

Hi @ChingizK, The issue you are experiencing might be because you are starting your path with a /. According to the provided information, when you enter the relative path, you should not begin it with / or ./.

For example, if the absolute path for the Python code you want to access is /python/covid_eda_raw.py, you should enter python/covid_eda_raw.py in the Path field.

So, in your case, instead of /Repos/.internal/4ac77871e5_commits/36c5bd9261b47a0bad5eaf99f32b9a2cd7032471/projects/membership-churn/mlops/workflow_notebooks_v1/02%20-%20Production%20Data%20ETL.py, you should use Repos/.internal/4ac77871e5_commits/36c5bd9261b47a0bad5eaf99f32b9a2cd7032471/projects/membership-churn/mlops/workflow_notebooks_v1/02%20-%20Production%20Data%20ETL.py.

Please adjust your path accordingly and try again.