brockb
Databricks Employee
Databricks Employee

Hi @Tommy ,

Thanks for your question.

I would encourage you to verify once using a Pro SQL Warehouse temporarily instead of a Serverless SQL Warehouse given the compute differences between the two - Pro compute resides in your data plane, Serverless compute is Databricks-managed. If it works as expected using a Pro Warehouse, there is a good indication that there is an issue with the network path to the Databricks-managed Serverless compute. If that is concluded, you can use docs such as this to guide you further on the Serverless setup: https://learn.microsoft.com/en-us/azure/databricks/admin/sql/serverless.

Additionally, Workspace-level SQL Warehouse configurations can be managed by a Workspace Admin via:

  • Settings / Workspace admin / Compute / and then clicking "Manage" next to "SQL warehouses and serverless compute".
  • This would be where you'd managed configs such as the `spark.conf.set("fs.azure.account...")`, as applicable, for all Warehouses in the workspace

Hope this helps