Databricks 13.3 Runtime Unable to use Python OS library to access mounted GCP bucket

[Kayla]

In Databricks 12.2, this Python code block lists out the contents of our ad_hoc folder in our mounted GCP bucket.

import os
os.listdir('/dbfs/mnt/hlm/ad_hoc/')

For some reason in 13.3 this same code block throws a " No such file or directory" error.

[Kaniz]

Hi @Kayla, The error "No such file or directory" indicates that the specified directory does not exist in the DBFS file system. This could be due to several reasons:

- The directory was deleted or moved.
- The directory name has changed.
- The mount point to the GCP bucket is not correctly configured or has been changed.
- The permissions to access the directory have changed. You can use the Databricks DBFS CLI or Databricks Runtime to verify the existence of the directory and the mount point. 

However, it's important to note that the Databricks CLI is unavailable on Databricks for GCP, as stated in the provided source. Therefore, you might want to use Databricks Runtime commands instead.

[Kayla · Accepted solution]

I've discovered the cause of this issue. The path is functional, but what I'm actually running into is a restriction due to the cluster set to "Shared" access mode:

• Cannot use R, RDD APIs, or clients that directly read the data from cloud storage, such as DBUtils.

Changing the cluster mode resolved the issue.