cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Accessing Workspace / Repo file works in notebook, but not from job

dollyb
Contributor

In a notebook attached to.a normal personal cluster I can successfully do this:

%fs ls file:/Workspace/Repos/$userName/$repoName/$folderName

When I run an init-script on a UC volume that the does the same thing, I'm getting this error:

ls: cannot access 'file:/Workspace/Repos/$userName/$repoName/$folderName': No such file or director

Both are run by the same user.

3 REPLIES 3

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @dollyb,

Within your init script are you doing %fs ls or unix command "ls" only? Looks like variables referenced on the path are not picking up! Have you tried adding some debug debug logic on your code? 

If you can DIM your init script I can have a look at it.

ameet9257
New Contributor III

Hi @dollyb ,

Can you please share some sample command which you are running in the init.sh file ? 

 

Amit Prajapati

dollyb
Contributor

It's simply the statement I listed in my post. There are no variables, I just wrote it like that to not post the exact values. So it's something like this in the script and that's it:

ls file:/Workspace/Repos/my_user_name@company.com/my_repo_name/my_folder_name

By the way I'm on Azure Databricks. Sometimes it seems this variant doesn't support all the features the AWS/GCP variants do.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group