topic Re: How we can send databricks log to Azure Application Insight ? in Data Engineering

How we can send databricks log to Azure Application Insight ?

Ajay-Pandey — Sat, 27 May 2023 18:07:44 GMT

Hi All,

I want to send databricks logs to azure application insight.

Is there any way we can do it ??

Any blog or doc will help me.

Re: How we can send databricks log to Azure Application Insight ?

Ajay-Pandey — Wed, 31 May 2023 12:05:05 GMT

#DAIS2023

Re: How we can send databricks log to Azure Application Insight ?

Ajay-Pandey — Mon, 05 Jun 2023 10:41:07 GMT

Thanks

Re: How we can send databricks log to Azure Application Insight ?

Debayan — Mon, 05 Jun 2023 07:16:19 GMT

Hi, You can follow : https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs.

Also please tag @Debayan with your next response which will notify me, Thank you!

Re: How we can send databricks log to Azure Application Insight ?

floringrigoriu — Fri, 08 Sep 2023 21:35:36 GMT

hi @Debayan in the https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs. there is a github repository mentioned https://github.com/mspnp/spark-monitoring ? That repository is marked as maintainance mode. Just want to double check if that is the latests resource we should use

Re: How we can send databricks log to Azure Application Insight ?

Debayan — Mon, 11 Sep 2023 06:42:34 GMT

Hi @floringrigoriu , As of now, yes that is the latest we can use. Thanks!

Re: How we can send databricks log to Azure Application Insight ?

madhavink14 — Fri, 08 Aug 2025 10:06:20 GMT

Hello @Debayan this code has been removed ..is there any documentation how we can send databricks jobs custom logs to application insight or log analytics workspace

Re: How we can send databricks log to Azure Application Insight ?

loic — Wed, 27 Aug 2025 13:59:57 GMT

Hello,
I am in the same case than you @madhavink14 .

The most recent post I found regarding what we want to achieve is:
https://dustinvannoy.com/2024/01/07/monitoring-databricks-with-log-analytics-updated/
It is based on this spark-monitoring repo:
https://github.com/dvannoy/spark-monitoring/tree/l4jv2_jars/target
Currently, by following this tutorial, I am able to have some log4j logs to be collected into a log Analytics workspace inside the sparkLogginEvent_CL table (spark-monitoring doesn't log to App Insights), but those logs have to come from a Notebook (tested with scala and python notebook).

So for now, logs that are done in a Jar file are not collected...

Re: How we can send databricks log to Azure Application Insight ?

loic — Wed, 12 Nov 2025 14:34:27 GMT

Hello,

I finally used he AppInsights agent from OpenTelemetry which is documented in the official Microsoft documentation here:

https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-enable?tabs=java

Below is an adaptation of this "Get started with OpenTelemetry" for an Azure Databricks compute.

1 - Upload the AppInsights.jar in the DBFS

2 - Make the jar to be uploaded in the compute.

From the settings of your compute, tab "configuration", then "advanced options", add an "init scripts" that will copy the jar on the compute.

This is what a really sample "init_script.sh" can looks like:
!/bin/bash
cp -f /dbfs/databricks/applicationinsights-agent-3.7.4.jar /databricks/jars

3 - Make the spark JVM to use the agent

This is achieved with a spark option that has to be applied to the driver and the executor (if you want to collect logs for both).

From the settings of your compute, tab "configuration", then "advanced options", add lines in "spark config".

spark.driver.extraJavaOptions -javaagent:/databricks/jars/applicationinsights-agent-3.7.4.jar

spark.executor.extraJavaOptions -javaagent:/databricks/jars/applicationinsights-agent-3.7.4.jar

4 - Configure the AppInsights agent to report to your AppInsights.

This is achieved with an environment variable (how to retrieve this connection string is already explained in the Microsoft link at the top of this post)

From the settings of your compute, tab "configuration", then "advanced options", add a line in "Spark : environment variables".

For example:

APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=XXXXXXXXXXXXXXX;IngestionEndpoint=https://XXXX.applicationinsights.azure.com/;LiveEndpoint=https://francecentral.livediagnostics.monitor.azure.com/;ApplicationId=XXXXXXXX

That's it, now your logs will be sent to the configured appInsights.

Of course, you can do advanced configuration for the AppInsights agent (agent role name, sampling etc..) by putting an applicationInsights.json file at the same location that the jar (adapt step 1 and 2 then)