topic Re: Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute in Data Engineering

Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute

Vetrivel — Tue, 15 Oct 2024 04:29:07 GMT

We have established an Azure Databricks workspace within our central subscription, which hosts all common platform resources. Additionally, we have a SQL Server running on a virtual machine in a separate sandbox subscription, containing data that needs to be accessed from Databricks.

We successfully configured VNet peering between the SQL VM's VNet and the Databricks VNet, allowing access to the SQL Server. Testing the connection from Databricks using a job cluster yielded successful results, confirming that everything functions as intended. However, when attempting to run the same notebook on a serverless compute, we encountered connectivity issues, preventing us from establishing a connection. This limitation compels us to rely on the shared job cluster, which requires time for scaling and initiating instances when none are active.

Your guidance on these matters would be greatly appreciated.

Re: Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute

Mo — Tue, 15 Oct 2024 09:38:46 GMT

hello @Vetrivel,

Serverless compute resources run in the serverless compute plane, which is managed by Azure Databricks. For more details on the network and architecture, see Serverless compute plane networking.

https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/

TL;DR: you need to use NCC to be able to access your resources.

Let me know if this helps.

Re: Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute

Vetrivel — Wed, 16 Oct 2024 05:58:02 GMT

Thanks @Mo . I will check and update.

Re: Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute

Vetrivel — Thu, 17 Oct 2024 05:58:00 GMT

@Mo I have tried and got below error.
Private access to resource type 'Microsoft.Compute/virtualMachines' is not supported with group id 'sqlserver'.I hope it supports only if the destinations are Blob, ADLS and Azure SQL.

Re: Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute

JeanBoutros — Mon, 11 Nov 2024 18:07:49 GMT

Have you had any luck with this? I'm facing the same problem.

Re: Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute

Vetrivel — Tue, 12 Nov 2024 07:16:24 GMT

No. I have got the attached reply from Microsoft Azure community.

Re: Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute

JeanBoutros — Mon, 18 Nov 2024 18:52:38 GMT

Thanks for that! I wish the migration to a managed instance was as easy 😄

have you looked into the proxy gateway option? I will ask my team maybe tomorrow if they have any experience with that.

Re: Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute

Vetrivel — Tue, 19 Nov 2024 04:22:01 GMT

No, we proceeded with the all purpose cluster.