Great use case!

Have you ever heard about Prometheus with Spark 3.0? Its a tool that can export live metrics for your jobs and runs which writes to a sink where you can read with a stream. I've personally never used in such use case, but there you can monitor every metric and write it out to then create some insights of it (such as averages and totals) on a different pipeline, which then can finally become a table.

To better understand, you can check these links below:

1. Session on how to use and enable Prometheus in databricks: https://www.youtube.com/watch?v=FDzm3MiSfiE

2. Spark official guide: https://spark.apache.org/docs/3.1.1/monitoring.html

Best,

Alessandro