ShamenParis
Contributor

Hi there!

Assuming your permissions are fully set up correctly, an indefinite hang (rather than a quick "403 Forbidden" error) usually points to either a timeout or a dropped network connection.

Here are three common culprits you might want to check out:

  • The folder has too many files: The Databricks UI is smart and uses pagination (it only loads the first 50–100 files at a time), which makes it load instantly. However, dbutils.fs.ls() is synchronous and tries to load the entire list of files into your cluster's memory all at once. If there are thousands of files in there, it will just hang. Try testing the command on a much smaller subfolder.

  • The missing trailing slash: Azure can be very picky about paths. If you run it without a slash at the end (.../foldername), Azure does a massive prefix scan just to check if "foldername" is a single file first. Try running it exactly like this with the slash at the end: .../foldername/ and see if it runs normally.

  • A hidden firewall/networking rule: The UI and your notebook actually run on different networks. The UI uses the Databricks Control Plane, while your notebook uses your cluster's specific VNet (the Data Plane). Your ADLS account might be set to "Allow Azure Services" (which lets the UI work), but a firewall or Network Security Group might be silently dropping the packets from your cluster's VNet, causing it to just hang while waiting for a response.

Hope one of these points you in the right direction!

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

View solution in original post