I know you can use microsoft diagnostics to receive notifications such as commandSubmit or commandFinish in json. But is it possible to get these diagnostics in databricks itself using pyspark?

I would like to get these diagnostics inside of a notebook instead of in azure.