topic Re: Azure Databricks Power Platform Connector - Doubts in Data Engineering

Azure Databricks Power Platform Connector - Doubts

JMartins777 — Tue, 01 Jul 2025 09:26:58 GMT

Hello,

Regarding the recently released azure databricks connector, i want to connect it to a Power App but i have 2 main questions which i need to know
1 - If the databricks URL is in a private network, how does it work and how can i achieve this connection.

2 - When i call this, it says it is real-time, but does it always call a cluster? How is it done, so i can get the data real-time without having to wait some minutes for the cluster to start?

Thanks in advance

Re: Azure Databricks Power Platform Connector - Doubts

Louis_Frolio — Tue, 01 Jul 2025 12:42:13 GMT

Connecting Power Apps to Azure Databricks in Private Networks

If your Azure Databricks workspace uses private networking (like VNet injection or Private Link), Power Apps cannot connect to it by default due to restrictive network access.
To allow Power Apps to reach your Databricks endpoint:
- You need to set up VNet peering or Private Link between the Databricks virtual network and the one that Power Platform uses.
- Make sure authentication is done using Microsoft Entra ID—either OAuth or a service principal.
- Once network connectivity is enabled and the right authentication is configured, you can add the Databricks connection in Power Apps.
If further restrictions exist (e.g., firewalls), you may need special components like an Azure Data Gateway, although this is more commonly documented for Power BI than Power Apps.

Real-Time Data Access and SQL Warehouse Behavior

The connector provides real-time data access by running queries on a Databricks SQL Warehouse, not by launching dedicated Spark clusters for each query.
If the SQL Warehouse is already running, results are returned quickly (seconds).
If the SQL Warehouse is paused or idle, the initial query after idling can take several minutes due to startup time. After it’s running, latency is minimal for subsequent queries.
You can minimize these delays by keeping the SQL Warehouse always running (using configuration to disable or extend auto-pause), but this increases compute cost.

Key Takeaways

For private networks: Ensure proper VNet peering/Private Link and use Entra ID authentication.
For real-time: All queries execute via SQL Warehouse; no Spark clusters are started. Keeping the SQL Warehouse active avoids cold-start delays.

Hope this helps, Lou.

Re: Azure Databricks Power Platform Connector - Doubts

nayan_wylde — Tue, 01 Jul 2025 14:39:54 GMT

1. Private Network Connectivity

For Databricks workspaces in private networks, you have a couple of options:

Option A: On-premises Data Gateway

Install the Microsoft On-premises Data Gateway in your private network
The gateway acts as a bridge between Power Apps (cloud) and your private Databricks workspace
Configure the gateway to route traffic through your private endpoints
This is the most common approach for private connectivity scenarios

Option B: Private Endpoints + VNet Integration

Azure Databricks supports browser authentication private endpoints for clients with no public internet connectivity Configure private connectivity to Azure Databricks - Azure Databricks | Microsoft Learn
Configure your Power Platform environment with VNet integration (if available in your tenant)
Ensure proper DNS resolution for private endpoint addresses
May require Power Platform Premium licensing for advanced networking features

Network Requirements:

Your private network must allow HTTPS (443) outbound to Power Platform services
Configure DNS to resolve Databricks private endpoint addresses
Ensure firewall rules allow the data gateway or VNet integration traffic

2. Real-time Performance & Cluster Management
The "real-time" capability is achieved through Databricks SQL Serverless, not traditional clusters
You can also try to put the cluster in some pool that you reduce cold start time.

topic Re: Azure Databricks Power Platform Connector - Doubts in Data Engineering

Azure Databricks Power Platform Connector - Doubts

Re: Azure Databricks Power Platform Connector - Doubts

Re: Azure Databricks Power Platform Connector - Doubts

1. Private Network Connectivity

2. Real-time Performance & Cluster Management