The error where files and directories can be read at the root ADLS level but not at the blob/subdirectory level, combined with a "No file or directory exists on path" message, is frequently due to permission configuration, incorrect path usage, or networking restrictions in Azure Databricks.
Troubleshooting Permissions and Access
-
Databricks may require "Storage Blob Data Reader" or "Contributor" roles at the storage account level, not just at the container or subdirectory level, especially when using Databricks Compute or serverless workloads.
-
Unity Catalog and workspace browsing may succeed with container-level permissions, but DBFS commands or direct SDK access from code (like your use of WorkspaceClient) may need broader account-level access due to how requests are made.
-
If multiple Service Principals are used, ensure the correct one with proper access is referenced.
-
Potential bugs or limitations exist where even with correct setup, Databricks fails to access subdirectories unless permissions are provided at the top storage account level.
Common Path and Configuration Issues
-
Double-check the abfss:// path syntax is correct. Ensure there are no typos or encoding errors; the path must be formatted exactly for ADLS Gen2.
-
The difference between mounting a container to DBFS and direct access can affect success. If not mounted, ensure Spark or DatabricksConnect configurations explicitly support direct ADLS access.
-
If using DBFS mounts (/mnt/*), check if remounting or restarting the cluster helps after rotating keys or permissions.
Network and Firewall Considerations
-
If the storage account restricts access to specific VNets, IPs, or uses Private Endpoints, the Databricks cluster must be correctly networked. Sometimes allowing traffic from all networks momentarily can confirm if this is the blocker.
-
Verify that the workspace, cluster, and storage account networking settings (VNet integration, service endpoints, firewalls) explicitly allow Databricks traffic.
Recommendations
-
Ensure "Storage Blob Data Reader" is assigned at the storage account level (not just to the container/subdirectory), at least for testing.
-
Validate the path and string encoding for your abfss://... references.
-
Cross-check which Service Principal or Managed Identity is being used; permissions must match.
-
If working with DBFS mounts, try remounting and restarting the cluster after role changes.
-
Review all networking and firewall settings if you suspect connectivity issues.
If these suggestions do not resolve the issue, it is possible you are encountering a limitation or bug with how Databricks Connect or the WorkspaceClient makes permission checks for blob-level paths, which may require raising a support ticket or checking with Databricks for known issues with container-level or subdirectory access
