Ashwin_DSA
Databricks Employee
Databricks Employee

Hi @NW1000,

I think you are seeing a permissions/identity difference between the two compute types, not a path or runtime issue.

  • On serverless interactive, the cluster runs as you, so it inherits your workspace permissions and can see everything under
    /Workspace/Users/xxx@xxxxxx.com/reporting/R/utils
    os.listdir shows your R files.

  • On your classic cluster, the code is likely running under a different principal (for example, a shared/standard cluster, or a job cluster “run as” a service principal). That identity either:

    1. doesn’t have read permission on your home folder, or
    2. only has very limited access there.

In that case, the directory itself exists (so os.path.exists is True), but the listing returns an empty result for that principal.

Try the below.

  1. Run on a single‑user (dedicated) cluster whose user is the same as the owner of /Workspace/Users/xxx@xxxxxx.com,
    or
  2. Move the files to a shared workspace folder and grant permissions.. and then point your code there to see if that helps address te problem.

 

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Regards,

Regards,
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***

View solution in original post