How we can send databricks log to Azure Application Insight ?

[Ajay-Pandey]

Hi All,

I want to send databricks logs to azure application insight.

Is there any way we can do it ??

Any blog or doc will help me.

[Ajay-Pandey]

#DAIS2023​ 

[Debayan]

Hi, You can follow : https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs.

Also please tag @Debayan​ with your next response which will notify me, Thank you!

[Ajay-Pandey]

Thanks

[floringrigoriu]

hi @Debayan in the  https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs. there is a github repository mentioned https://github.com/mspnp/spark-monitoring ? That repository is marked as  maintainance mode.  Just want to double check if that is the latests resource we should use 

[Debayan]

Hi @floringrigoriu , As of now, yes that is the latest we can use. Thanks!

[madhavink14]

Hello @Debayan  this code has been removed ..is there any documentation how we can send databricks jobs custom logs to application insight or log analytics workspace

[loic]

Hello,
I am in the same case than you @madhavink14 .

The most recent post I found regarding what we want to achieve is:
https://dustinvannoy.com/2024/01/07/monitoring-databricks-with-log-analytics-updated/
It is based on this spark-monitoring repo:
https://github.com/dvannoy/spark-monitoring/tree/l4jv2_jars/target
Currently, by following this tutorial, I am able to have some log4j logs to be collected into a log Analytics workspace inside the sparkLogginEvent_CL table (spark-monitoring doesn't log to App Insights), but those logs have to come from a Notebook (tested with scala and python notebook).

So for now, logs that are done in a Jar file are not collected...