I've read that in the documentation, and when I now tried with an Access Connector for Azure Databricks instead of my own service principal, it seems to have worked, shockingly, even if I completely block network access on the storage account with zero private endpoints. No idea how, but for anyone coming across this, the solution is to use an Access Connector for Azure Databricks.