Hi @saurabh18cs , thank you for your response. 

Could you please share documentation on the first method you recommended: "Enable Spark Metrics: Databricks provides detailed metrics for Spark jobs, stages, and tasks. You can enable these metrics and send them to Azure Log Analytics."?

I am a bit unsure about the usage of this, so some reference material would really help!