Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute

[Vetrivel]

We have established an Azure Databricks workspace within our central subscription, which hosts all common platform resources. Additionally, we have a SQL Server running on a virtual machine in a separate sandbox subscription, containing data that needs to be accessed from Databricks.

We successfully configured VNet peering between the SQL VM's VNet and the Databricks VNet, allowing access to the SQL Server. Testing the connection from Databricks using a job cluster yielded successful results, confirming that everything functions as intended. However, when attempting to run the same notebook on a serverless compute, we encountered connectivity issues, preventing us from establishing a connection. This limitation compels us to rely on the shared job cluster, which requires time for scaling and initiating instances when none are active.

Your guidance on these matters would be greatly appreciated.

[Mo]

hello @Vetrivel,

Serverless compute resources run in the serverless compute plane, which is managed by Azure Databricks. For more details on the network and architecture, see Serverless compute plane networking.

https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/

TL;DR: you need to use NCC to be able to access your resources.

Let me know if this helps.

[Vetrivel]

Thanks @Mo . I will check and update.