@Saurabh2406  this is such a rich article and has so many practical takeaways! Congrats!

I faced similar challenges in one of my last projects, and I could spend some time building a nice dashboard (using the system.billing tables) that helped us track the estimated total processing cost in Databricks per data pipeline and dataset (looking also at how the cost was evolving and being affected by corrective actions). That helped us find the least efficient processes and start optimizing from there.

Can you share some ideas on which type of metrics you found useful in your experience?