Use Python code from a remote Git repository

Data Engineering

I'm trying to create a task where the source is a Python script located in remote GitLab repo. I'm following the instructions HERE and this is how I have the task set up:

However, no matter what path I specify all I get is the error below:

Cannot read the python file /Repos/.internal/4ac77871e5_commits/36c5bd9261b47a0bad5eaf99f32b9a2cd7032471/projects/membership-churn/mlops/workflow_notebooks_v1/02%20-%20Production%20Data%20ETL.py. Please check driver logs for more details

I've searched the Community and found THIS and tried to follow the advice mentioned there.

The Logs show the below AssertionError:

23/09/21 19:25:24 WARN JupyterDriverLocal: User code returned error with traceback: [0;31m---------------------------------------------------------------------------[0m
[0;31mAssertionError[0m                            Traceback (most recent call last)
File [0;32m~/.ipykernel/1132/command--1-426573936:2[0m
[1;32m      1[0m [38;5;28;01mfrom[39;00m [38;5;21;01mos[39;00m[38;5;21;01m.[39;00m[38;5;21;01mpath[39;00m [38;5;28;01mimport[39;00m exists
[0;32m----> 2[0m [38;5;28;01massert[39;00m(exists([38;5;124m"[39m[38;5;124m/Workspace/Repos/.internal/4ac77871e5_commits/36c5bd9261b47a0bad5eaf99f32b9a2cd7032471/projects/membership-churn/mlops/workflow_notebooks_v1/02[39m[38;5;124m%[39m[38;5;124m20-[39m[38;5;124m%[39m[38;5;124m20Production[39m[38;5;124m%[39m[38;5;124m20Data[39m[38;5;132;01m%20E[39;00m[38;5;124mTL.py[39m[38;5;124m"[39m))

I'm not quite sure why is it trying to invoke a Jupyter Driver since the files in my repo are all *.py

What am I doing wrong here?

1 REPLY 1

Hi @ChingizK, The issue you are experiencing might be because you are starting your path with a /. According to the provided information, when you enter the relative path, you should not begin it with / or ./.

For example, if the absolute path for the Python code you want to access is /python/covid_eda_raw.py, you should enter python/covid_eda_raw.py in the Path field.

So, in your case, instead of /Repos/.internal/4ac77871e5_commits/36c5bd9261b47a0bad5eaf99f32b9a2cd7032471/projects/membership-churn/mlops/workflow_notebooks_v1/02%20-%20Production%20Data%20ETL.py, you should use Repos/.internal/4ac77871e5_commits/36c5bd9261b47a0bad5eaf99f32b9a2cd7032471/projects/membership-churn/mlops/workflow_notebooks_v1/02%20-%20Production%20Data%20ETL.py.

Please adjust your path accordingly and try again.