How we can send databricks log to Azure Application Insight ?

[Ajay-Pandey]

Hi All,

I want to send databricks logs to azure application insight.

Is there any way we can do it ??

Any blog or doc will help me.

[Ajay-Pandey]

#DAIS2023​ 

[Debayan]

Hi, You can follow : https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs.

Also please tag @Debayan​ with your next response which will notify me, Thank you!

[Ajay-Pandey]

Thanks

[floringrigoriu]

hi @Debayan in the  https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs. there is a github repository mentioned https://github.com/mspnp/spark-monitoring ? That repository is marked as  maintainance mode.  Just want to double check if that is the latests resource we should use 

[Debayan]

Hi @floringrigoriu , As of now, yes that is the latest we can use. Thanks!

[madhavink14]

Hello @Debayan  this code has been removed ..is there any documentation how we can send databricks jobs custom logs to application insight or log analytics workspace

[loic]

Hello,
I am in the same case than you @madhavink14 .

The most recent post I found regarding what we want to achieve is:
https://dustinvannoy.com/2024/01/07/monitoring-databricks-with-log-analytics-updated/
It is based on this spark-monitoring repo:
https://github.com/dvannoy/spark-monitoring/tree/l4jv2_jars/target
Currently, by following this tutorial, I am able to have some log4j logs to be collected into a log Analytics workspace inside the sparkLogginEvent_CL table (spark-monitoring doesn't log to App Insights), but those logs have to come from a Notebook (tested with scala and python notebook).

So for now, logs that are done in a Jar file are not collected...

[loic]

Hello,

I finally used he AppInsights agent from OpenTelemetry which is documented in the official Microsoft documentation here:

https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-enable?tabs=java

Below is an adaptation of this "Get started with OpenTelemetry" for an Azure Databricks compute.

1 - Upload the AppInsights.jar in the DBFS

2 - Make the jar to be uploaded in the compute.

From the settings of your compute, tab "configuration", then "advanced options", add an "init scripts" that will copy the jar on the compute.

This is what a really sample "init_script.sh" can looks like:
!/bin/bash
cp -f /dbfs/databricks/applicationinsights-agent-3.7.4.jar /databricks/jars

3 - Make the spark JVM to use the agent

This is achieved with a spark option that has to be applied to the driver and the executor (if you want to collect logs for both).

From the settings of your compute, tab "configuration", then "advanced options", add lines in "spark config".

spark.driver.extraJavaOptions -javaagent:/databricks/jars/applicationinsights-agent-3.7.4.jar

spark.executor.extraJavaOptions -javaagent:/databricks/jars/applicationinsights-agent-3.7.4.jar

4 - Configure the AppInsights agent to report to your AppInsights.

This is achieved with an environment variable (how to retrieve this connection string is already explained in the Microsoft link at the top of this post)

From the settings of your compute, tab "configuration", then "advanced options", add a line in "Spark : environment variables".

For example:

APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=XXXXXXXXXXXXXXX;IngestionEndpoint=https://XXXX.applicationinsights.azure.com/;LiveEndpoint=https://francecentral.livediagnostics.monit...

That's it, now your logs will be sent to the configured appInsights.

Of course, you can do advanced configuration for the AppInsights agent (agent role name, sampling etc..) by putting an applicationInsights.json file at the same location that the jar (adapt step 1 and 2 then)

More information on this topic here:

https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config

Regards,

Loïc