How to get cluster metrics by Power BI?

[Mado]

Assume that I have a few clusters in Databricks and I want to make a report in Power BI that shows cluster metrics. For example:

• How many hours clusters were running categorized by username
• Cost
• CPU load

Is there any way that I can send this information from Databricks to Power BI?

Also, I want it to work continuously (not only one time).

[Anonymous · Accepted solution]

@Mohammad Saber​ :

Here's an overview of how you can set up a pipeline to send cluster metrics from Databricks to Power BI:

Configure the Databricks cluster to send logs to an Azure Event Hub or Azure Log Analytics workspace. You can do this by following the instructions in the Databricks documentation:

• https://docs.databricks.com/administration-guide/monitoring/cloud-watch.html
  • https://docs.databricks.com/administration-guide/monitoring/log-analytics.html

Create an Azure Stream Analytics job that reads the logs from the Event Hub or Log Analytics workspace, transforms the data as needed, and sends it to Power BI. You can use the Azure Stream Analytics portal to create the job and define the input, query, and output. You can find more information on how to create a Stream Analytics job in the Azure documentation:

• https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-quick-create-portal

In Power BI, create a new report that connects to the Stream Analytics output and shows the desired metrics. You can use the Power BI Desktop or the Power BI service to create the report. You can find more information on how to create a report in the Power BI documentation:

• https://docs.microsoft.com/en-us/power-bi/create-reports/desktop-quickstart-report-tutorial

Once you have set up the pipeline, the data will flow continuously from Databricks to Power BI, and you can monitor the cluster metrics in real-time.

[Anonymous]

Hi @Mohammad Saber​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!