โ10-14-2024 09:28 PM - edited โ10-14-2024 09:29 PM
We have established an Azure Databricks workspace within our central subscription, which hosts all common platform resources. Additionally, we have a SQL Server running on a virtual machine in a separate sandbox subscription, containing data that needs to be accessed from Databricks.
We successfully configured VNet peering between the SQL VM's VNet and the Databricks VNet, allowing access to the SQL Server. Testing the connection from Databricks using a job cluster yielded successful results, confirming that everything functions as intended. However, when attempting to run the same notebook on a serverless compute, we encountered connectivity issues, preventing us from establishing a connection. This limitation compels us to rely on the shared job cluster, which requires time for scaling and initiating instances when none are active.
Your guidance on these matters would be greatly appreciated.
โ10-15-2024 02:38 AM
hello @Vetrivel,
Serverless compute resources run in the serverless compute plane, which is managed by Azure Databricks. For more details on the network and architecture, see Serverless compute plane networking.
https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/
TL;DR: you need to use NCC to be able to access your resources.
Let me know if this helps.
โ10-15-2024 10:58 PM
Thanks @Mo . I will check and update.
โ10-16-2024 10:58 PM
@Mo I have tried and got below error.
Private access to resource type 'Microsoft.Compute/virtualMachines' is not supported with group id 'sqlserver'.I hope it supports only if the destinations are Blob, ADLS and Azure SQL.
3 weeks ago
Have you had any luck with this? I'm facing the same problem.
3 weeks ago
2 weeks ago
Thanks for that! I wish the migration to a managed instance was as easy ๐
have you looked into the proxy gateway option? I will ask my team maybe tomorrow if they have any experience with that.
2 weeks ago
No, we proceeded with the all purpose cluster.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group