Azure Databricks Power Platform Connector - Doubts

[JMartins777]

Hello, 

Regarding the recently released azure databricks connector, i want to connect it to a Power App but i have 2 main questions which i need to know
1 - If the databricks URL is in a private network, how does it work and how can i achieve this connection.

2 - When i call this, it says it is real-time, but does it always call a cluster? How is it done, so i can get the data real-time without having to wait some minutes for the cluster to start?

Thanks in advance

[BigRoux · Accepted solution]

Connecting Power Apps to Azure Databricks in Private Networks

• If your Azure Databricks workspace uses private networking (like VNet injection or Private Link), Power Apps cannot connect to it by default due to restrictive network access.
• To allow Power Apps to reach your Databricks endpoint:
  • You need to set up VNet peering or Private Link between the Databricks virtual network and the one that Power Platform uses.
  • Make sure authentication is done using Microsoft Entra ID—either OAuth or a service principal.
  • Once network connectivity is enabled and the right authentication is configured, you can add the Databricks connection in Power Apps.
• If further restrictions exist (e.g., firewalls), you may need special components like an Azure Data Gateway, although this is more commonly documented for Power BI than Power Apps.

Real-Time Data Access and SQL Warehouse Behavior

• The connector provides real-time data access by running queries on a Databricks SQL Warehouse, not by launching dedicated Spark clusters for each query.
• If the SQL Warehouse is already running, results are returned quickly (seconds).
• If the SQL Warehouse is paused or idle, the initial query after idling can take several minutes due to startup time. After it’s running, latency is minimal for subsequent queries.
• You can minimize these delays by keeping the SQL Warehouse always running (using configuration to disable or extend auto-pause), but this increases compute cost.

Key Takeaways

• For private networks: Ensure proper VNet peering/Private Link and use Entra ID authentication.
• For real-time: All queries execute via SQL Warehouse; no Spark clusters are started. Keeping the SQL Warehouse active avoids cold-start delays.

Hope this helps, Lou.

[nayan_wylde]

1. Private Network Connectivity

For Databricks workspaces in private networks, you have a couple of options:

Option A: On-premises Data Gateway

• Install the Microsoft On-premises Data Gateway in your private network
• The gateway acts as a bridge between Power Apps (cloud) and your private Databricks workspace
• Configure the gateway to route traffic through your private endpoints
• This is the most common approach for private connectivity scenarios

Option B: Private Endpoints + VNet Integration

• Azure Databricks supports browser authentication private endpoints for clients with no public internet connectivity Configure private connectivity to Azure Databricks - Azure Databricks | Microsoft Learn
• Configure your Power Platform environment with VNet integration (if available in your tenant)
• Ensure proper DNS resolution for private endpoint addresses
• May require Power Platform Premium licensing for advanced networking features

Network Requirements:

• Your private network must allow HTTPS (443) outbound to Power Platform services
• Configure DNS to resolve Databricks private endpoint addresses
• Ensure firewall rules allow the data gateway or VNet integration traffic
  
  
  

  2. Real-time Performance & Cluster Management

  The "real-time" capability is achieved through Databricks SQL Serverless, not traditional clusters

• You can also try to put the cluster in some pool that you reduce cold start time.