How to trigger Power BI refresh from Databricks pipeline without keeping cluster alive?

[chandataeng]

I have a Databricks pipeline that pulls data from AWS, which takes ~90 minutes. After this, I need to refresh a series of Power BI dataflows (~45 mins) and then datasets (~45 mins).

I want to trigger the Power BI refresh automatically from Databricks once the pipeline finishes. However, if I run this as a final task in the Databricks job, the cluster has to stay alive for up to 90 extra minutes while Power BI refreshes — which wastes compute and costs.

I want to avoid scheduled refreshes in Power BI since they don’t align well with my pipeline timing.

Is there a recommended architecture to:

• Trigger Power BI refreshes programmatically from Databricks

• Let the refresh continue independently after triggering

• Avoid cluster runtime during Power BI refresh wait time?

[szymon_dybczak · Accepted solution]

Hi @chandataeng ,

The current Power BI task that is available in databricks workflow will wait for refresh process to return correct status (whether it succeeded or failed).

But you can start refresh process by using asynchronous REST API call. The refresh process will start and your cluster can be terminated after that. I'm using this approach in one of the client I worked with.

Enhanced refresh with the Power BI REST API - Power BI | Microsoft Learn