How to access Spark UI metrics in an automated way (API)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-28-2024 10:57 AM
I am doing some automated testing; and would like ultimately to access per job/stage/task metrics as shown in the UI (e.g. spark UI -> sql dataframe) -> plan visualization in an automated way (API is ideal; but some ad-hoc metrics pipelines from local -> s3 also solves the problem).
I have followed this tutorial (https://kb.databricks.com/clusters/persist-metrics-csv-sink-dbfs) to set up metrics; but these are raw metrics arranged by timestamp; without any breakdown e.g. by tasks etc.
It seems I may need to enable eventLog; but when I set the following; my cluster fails to come up:
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-28-2024 02:45 PM
Thanks for the response. This enables the event logs. But the event logs seem to be empty. Would you know where I can get the spark metrics as seen from the spark ui.