Dan_Z
Databricks Employee
Databricks Employee

@Franklin George​ , Honestly, there is no easy way to do this. Your only option is to set up cluster log delivery, which will give you access to the cluster's event log file. This event log file is JSON and contains all of the info that the SparkUI uses (and more). It will have the information you are looking for but is not trivial to parse manually. I can't think of a better option.

View solution in original post