How to access Spark UI metrics in an automated way (API)

[superspan]

I am doing some automated testing; and would like ultimately to access per job/stage/task metrics as shown in the UI (e.g. spark UI -> sql dataframe) -> plan visualization in an automated way (API is ideal; but some ad-hoc metrics pipelines from local -> s3 also solves the problem).

I have followed this tutorial (https://kb.databricks.com/clusters/persist-metrics-csv-sink-dbfs) to set up metrics; but these are raw metrics arranged by timestamp; without any breakdown e.g. by tasks etc.

It seems I may need to enable eventLog; but when I set the following; my cluster fails to come up:

spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
 

[Kaniz]

Hi @superspan, 

• Event logs provide detailed information about Spark application execution, including task-level data.

• Unfortunately, directly configuring spark.eventLog.dir in a Databricks notebook won’t work. Instead, follow these steps:

• Step 1: Go to your Databricks workspace and navigate to the Clusters section.

• Step 2: Select the cluster you’re interested in and click the Edit button.

• Step 3: Under Advanced Options, click on Spark, then Spark Config.

• Step 4: Add the following configurations:

  • spark.eventLog.enabled true
  • spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/

• Step 5: Confirm your changes and restart the cluster.

[superspan]

Thanks for the response. This enables the event logs. But the event logs seem to be empty. Would you know where I can get the spark metrics as seen from the spark ui.