topic Re: Accessing Workspace / Repo file works in notebook, but not from job in Data Engineering

Accessing Workspace / Repo file works in notebook, but not from job

dollyb — Thu, 19 Dec 2024 12:07:40 GMT

In a notebook attached to.a normal personal cluster I can successfully do this:

%fs ls file:/Workspace/Repos/$userName/$repoName/$folderName

When I run an init-script on a UC volume that the does the same thing, I'm getting this error:

ls: cannot access 'file:/Workspace/Repos/$userName/$repoName/$folderName': No such file or director

Both are run by the same user.

Re: Accessing Workspace / Repo file works in notebook, but not from job

Alberto_Umana — Thu, 19 Dec 2024 12:52:21 GMT

Hi @dollyb,

Within your init script are you doing %fs ls or unix command "ls" only? Looks like variables referenced on the path are not picking up! Have you tried adding some debug debug logic on your code?

If you can DIM your init script I can have a look at it.

Re: Accessing Workspace / Repo file works in notebook, but not from job

ameet9257 — Thu, 19 Dec 2024 13:33:59 GMT

Hi @dollyb ,

Can you please share some sample command which you are running in the init.sh file ?

Re: Accessing Workspace / Repo file works in notebook, but not from job

dollyb — Thu, 19 Dec 2024 16:28:26 GMT

It's simply the statement I listed in my post. There are no variables, I just wrote it like that to not post the exact values. So it's something like this in the script and that's it:

ls file:/Workspace/Repos/my_user_name@company.com/my_repo_name/my_folder_name

By the way I'm on Azure Databricks. Sometimes it seems this variant doesn't support all the features the AWS/GCP variants do.

Re: Accessing Workspace / Repo file works in notebook, but not from job

Alberto_Umana — Thu, 19 Dec 2024 19:23:55 GMT

Hi @dollyb,

Can you try with just "ls /Workspace/Repos/my_user_name@company.com/my_repo_name/my_folder_name"

I'm not sure dbutils will be useful in an init script, I will try to test it out

Re: Accessing Workspace / Repo file works in notebook, but not from job

dollyb — Fri, 20 Dec 2024 06:00:19 GMT

I'm not using dbutils in my init-script, just the unix ls command.