A capability that often goes unnoticed in Databricks Workflows is the Power BI task type. After exploring this feature in detail, it becomes clear that a common challenge faced by many data teams can now be solved more efficiently: Power BI semantic models can be refreshed directly from Databricks pipelines.
The Traditional Workflow:
In most modern analytics architectures, data transformation is performed inside Databricks pipelines. Once the transformations are completed and tables are updated, the Power BI dataset refresh must usually be triggered separately.
This refresh process is typically handled through:
- Scheduled refreshes inside Power BI
- External API calls
- Additional orchestration tools
Although these approaches work, they introduce extra dependencies and operational complexity into the workflow.
A Simpler Approach with the Power BI Task:
The introduction of the Power BI task within Databricks Workflows simplifies this process significantly.
Once a pipeline completes its data transformations, a Power BI task can be added directly to the workflow. The semantic model can then be refreshed automatically as part of the same pipeline execution.
Because of this integration:
- External scripts are no longer required
- Refresh orchestration can be handled within Databricks itself
- Data engineering pipelines and BI refresh processes can be managed together
This results in a more streamlined and unified data pipeline.
Integration with Unity Catalog:
Another key aspect of this feature is its native integration with Unity Catalog.
Power BI semantic models can directly consume data objects stored in Unity Catalog, including:
- Tables
- Views
- Materialized views
- Streaming tables
As a result, curated and governed datasets maintained in Databricks can be used directly as the foundation for Power BI semantic models.
Even more flexibility is provided because data can be combined across multiple catalogs and schemas within Unity Catalog. This is particularly valuable in environments where data domains are distributed across multiple sources.
Security and Governance:
From a governance standpoint, the integration relies on Microsoft Entra ID authentication using OAuth and Single Sign-On (SSO).
Because of this architecture:
- Unity Catalog permissions can be respected within Power BI queries.
- Access policies remain consistent across both platforms.
- Data governance rules are enforced throughout the analytics stack.
This ensures that security controls remain centralized and consistent.
Smarter Dataset Refresh:
Another advantage of integrating Power BI refresh with Databricks workflows is that refresh operations can be triggered only when data changes occur.
Instead of refreshing datasets on fixed schedules, refresh tasks can be executed after upstream data pipelines complete successfully.
This approach can:
- Reduce unnecessary refresh operations.
- Improve pipeline efficiency.
- Ensure dashboards always reflect the most recent data.
Orchestrating Power BI from Databricks:
Although Power BI datasets can be published manually from Databricks, the Power BI task enables automatic orchestration of semantic models directly within workflows.
Before configuring this integration, the same prerequisites required for manual publishing must be satisfied.
Step 1: Preparing Authentication Credentials
To allow Databricks to interact with Power BI, authentication credentials must first be prepared.
Two authentication methods are supported:
- OAuth Machine-to-Machine
- OAuth User-to-Machine
OAuth Machine-to-Machine Setup
For machine-to-machine authentication, the following steps must be completed:
- An Azure service principal must be created or an existing one must be used through the Azure portal.
- The service principal must be added to the Power BI workspace where datasets are published.
- Service principals must be enabled in the Power BI Admin Portal.
- A client secret must be generated, and both the client ID and client secret must be stored.
- The authorization endpoint must be recorded, which follows this format:
https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/authorize.
OAuth User-to-Machine Setup
For user-based authentication, an authorization endpoint must also be obtained.
The endpoint follows the same structure:
https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/authorize
Here, {tenant_id} represents the Microsoft Entra tenant associated with the organization.
If the dataset resides inside My Workspace (personal workspace), the tenant ID can be replaced with:
https://login.microsoftonline.com/common/oauth2/v2.0/authorize
When the connection is created, a login window will be displayed. The user will sign in and grant access to an Entra application named:
Databricks Dataset Publishing Integration
This application enables Databricks to interact with Power BI resources on behalf of the user.
Step 2: Creating a Power BI Connection in Unity Catalog:
To orchestrate Power BI tasks, a connection must first be created inside Unity Catalog.
Note: The CREATE CONNECTION privilege is required to create this connection. If it is created by another user, the USE CONNECTION privilege must be granted.
- The Databricks workspace should be opened and Catalog Explorer should be accessed.
- The Create → Create connection option should be selected.
- A connection name should be provided.
- The connection type should be set to Power BI.
- For Auth type, select OAuth Machine to Machine or OAuth (user to machine), based on the credentials you prepared previously and Click Next
- In Step 2 of the Set up connection wizard, you are prompted for the credential information you prepared.
- Login with your PowerBI workspace credentials.
Once successfully logged in you can create the connection
- Click Create connection
Once the connection is created, the below page appears. This connection can now be used when configuring a Power BI task.
If access needs to be shared with other users, the USE CONNECTION privilege can be granted.
Step 3: Loading Data into Databricks:
- Before refreshing a Power BI semantic model, a data ingestion or transformation job must be created in Databricks.
This job is responsible for:
i. Extracting data from the source system.
ii. Transforming the data.
iii. Loading it into curated tables within Databricks - Once the data is prepared, Power BI dashboards can be built on top of these datasets.
Step 4: Creating a Power BI Task in Databricks Jobs:
To automate dataset refresh, a Power BI task must be added to a Databricks job workflow.
- The Jobs UI should be opened.
- Click + Add task.
- Enter a Task name.
- In the Type drop-down menu, select Power BI
- Once Power BI is selected, a Power BI connection must be chosen, the connection that was created earlier should be selected.
If a connection has not been created previously, click on the New Connection option and Step 2 should be followed to create the required connection. - After selecting the connection, additional properties must be configured:
- The Power BI workspace containing the dashboard
- The semantic model(dataset)
- The query mode
-
The tables that require refresh
- Select the Power BI task created earlier configured to depend on the upstream data loading job, ensuring the refresh only runs after data processing completes.
- The Power BI workspace containing the dashboard
- Click Save task.
Step 5: Triggering the Dataset Refresh:
Once the workflow has been configured, the refresh process can be triggered.
- Click Run now, to trigger the refresh, the pipeline will execute the data transformation job first, followed by the Power BI task, which refreshes the semantic model automatically.
As a result, the Power BI dataset in the workspace will be updated with the latest data generated by the pipeline.
Final Thoughts:
The Power BI task in Databricks represents an important step toward a fully integrated analytics architecture.
Instead of managing data pipelines and BI refresh schedules separately, both processes can now be orchestrated within a single workflow.
This integration simplifies pipeline management, improves data freshness, and reduces operational complexity. For teams already using Databricks for data engineering and Power BI for analytics, this feature can significantly streamline the end-to-end data workflow.
1 Comment
