mzs
Contributor

The problem was actually with DBFS and the internal Databricks-managed storage account firewall, not even with the storage account my catalog is using. The cluster event logs would occasionally show "DBFS is down".

In Terraform, in my azurerm_databricks_workspace resource, I had set default_storage_firewall_enabled = true. This sets up the firewall on the internal storage account and adds subnets from NCC, but not classic compute subnets. To make that work I would need to set up private endpoints for the internal storage account: https://learn.microsoft.com/en-us/azure/databricks/security/network/storage/firewall-support

Since we don't have anything using DBFS explicitly, I turned off DBFS in workspace security settings ("Disable DBFS root and mounts"), and now I'm able to work with files and tables from Unity Catalog in a notebook. The "DBFS is down" messages are gone from the event log as well.

View solution in original post