I am doing some automated testing; and would like ultimately to access per job/stage/task metrics as shown in the UI (e.g. spark UI -> sql dataframe) -> plan visualization in an automated way (API is ideal; but some ad-hoc metrics pipelines from local -> s3 also solves the problem).
I have followed this tutorial (https://kb.databricks.com/clusters/persist-metrics-csv-sink-dbfs) to set up metrics; but these are raw metrics arranged by timestamp; without any breakdown e.g. by tasks etc.
It seems I may need to enable eventLog; but when I set the following; my cluster fails to come up:
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/