cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Accessing Workspace / Repo file works in notebook, but not from job

dollyb
Contributor

In a notebook attached to.a normal personal cluster I can successfully do this:

%fs ls file:/Workspace/Repos/$userName/$repoName/$folderName

When I run an init-script on a UC volume that the does the same thing, I'm getting this error:

ls: cannot access 'file:/Workspace/Repos/$userName/$repoName/$folderName': No such file or director

Both are run by the same user.

5 REPLIES 5

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @dollyb,

Within your init script are you doing %fs ls or unix command "ls" only? Looks like variables referenced on the path are not picking up! Have you tried adding some debug debug logic on your code? 

If you can DIM your init script I can have a look at it.

ameet9257
Contributor

Hi @dollyb ,

Can you please share some sample command which you are running in the init.sh file ? 

 

Amit Prajapati

dollyb
Contributor

It's simply the statement I listed in my post. There are no variables, I just wrote it like that to not post the exact values. So it's something like this in the script and that's it:

ls file:/Workspace/Repos/my_user_name@company.com/my_repo_name/my_folder_name

By the way I'm on Azure Databricks. Sometimes it seems this variant doesn't support all the features the AWS/GCP variants do.

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @dollyb,

Can you try with just "ls /Workspace/Repos/my_user_name@company.com/my_repo_name/my_folder_name"

I'm not sure dbutils will be useful in an init script, I will try to test it out

I'm not using dbutils in my init-script, just the unix ls command.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now