get diagnostics using pyspark instead of microsoft diagnostics

[justworking]

I know you can use microsoft diagnostics to receive notifications such as commandSubmit or commandFinish in json. But is it possible to get these diagnostics in databricks itself using pyspark?

I would like to get these diagnostics inside of a notebook instead of in azure.

[Kaniz_Fatma]

Hi @hamza o​ , Azure Databricks provides comprehensive end-to-end diagnostic logs of activities performed by Azure Databricks users, allowing your enterprise to monitor detailed Azure Databricks usage patterns.

For a list of each of these types of events and the associated services, see Events. Some of the events are emitted in audit logs only if verbose audit logs are enabled for the workspace.

In addition to the default events, you can configure a workspace to generate additional events by enabling verbose audit logs.

• https://learn.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagn...