Hi @CG29,

I suspect the main thing missing here is that these two access paths aren't always using the same permission model. When you browse the path from the Databricks UI as an external location, that access is typically governed through Unity Catalog using an external location plus its storage credential, whereas a direct dbutils.fs.ls("abfss://...") call is raw path-based access to cloud storage and therefore depends on the permissions available for that direct URI access path. Databricks calls out both patterns in the public docs and generally recommends using Unity Catalog-managed access where possible rather than relying on raw cloud URIs for day-to-day access. See Work with files on Azure Databricks, Connect to cloud object storage using Unity Catalog, and Connect to an Azure Data Lake Storage Gen2 external location.

So if the container root lists successfully but abfss://.../foldername hangs, the most likely explanation is that the identity used for the direct abfss call can see the container itself but lacks sufficient ADLS Gen2 permissions on that subdirectory or one of its parent paths. In ADLS Gen2, listing a directory requires read and execute permissions on that directory, and traversing into nested paths requires execute permissions on the parent directories as well. Microsoft documents that behaviour in the ACL guidance here: Access control lists in Azure Data Lake Storage.

In other words, being able to see the location in the Databricks UI does not necessarily prove that the same notebook call over abfss:// has the required permissions end-to-end. I would first verify which identity is actually being used by the cluster for direct ADLS access, and then check the ACLs on foldername and its parent directories to make sure that identity has the required traverse and list permissions. If ACLs were only set at a higher level, it is also worth checking whether they were ever propagated recursively to existing child folders and files. Microsoft’s ADLS ACL docs cover that as well in the CLI guidance for recursive ACL updates.

It is also worth trying the path with a trailing slash, since Databricks examples for directory listings use that form, for example dbutils.fs.ls("abfss://container@account.dfs.core.windows.net/path/"). That probably isn’t the root cause here, but it is a quick sanity check.

Hope this helps.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.