https://community.databricks.com/t5/data-engineering/how-to-access-spark-ui-metrics-in-an-automated-way-api/m-p/64973#M32717 <P>Thanks for the response. This enables the event logs. But the event logs seem to be empty. Would you know where I can get the spark metrics as seen from the spark ui.</P> Thu, 28 Mar 2024 21:45:23 GMT superspan 2024-03-28T21:45:23Z https://community.databricks.com/t5/data-engineering/how-to-access-spark-ui-metrics-in-an-automated-way-api/m-p/64947#M32708 <P>I am doing some automated testing; and would like ultimately to access per job/stage/task metrics as shown in the UI (e.g. spark UI -&gt; sql dataframe) -&gt; plan visualization in an automated way (API is ideal; but some ad-hoc metrics pipelines from local -&gt; s3 also solves the problem).</P><P>I have followed this tutorial (<A href="https://kb.databricks.com/clusters/persist-metrics-csv-sink-dbfs" target="_blank">https://kb.databricks.com/clusters/persist-metrics-csv-sink-dbfs</A>) to set up metrics; but these are raw metrics arranged by timestamp; without any breakdown e.g. by tasks etc.</P><P>It seems I may need to enable eventLog; but when I set the following; my cluster fails to come up:</P><P>spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/<BR />&nbsp;</P> Thu, 28 Mar 2024 17:57:12 GMT https://community.databricks.com/t5/data-engineering/how-to-access-spark-ui-metrics-in-an-automated-way-api/m-p/64947#M32708 superspan 2024-03-28T17:57:12Z https://community.databricks.com/t5/data-engineering/how-to-access-spark-ui-metrics-in-an-automated-way-api/m-p/64973#M32717 <P>Thanks for the response. This enables the event logs. But the event logs seem to be empty. Would you know where I can get the spark metrics as seen from the spark ui.</P> Thu, 28 Mar 2024 21:45:23 GMT https://community.databricks.com/t5/data-engineering/how-to-access-spark-ui-metrics-in-an-automated-way-api/m-p/64973#M32717 superspan 2024-03-28T21:45:23Z