loic
Contributor

‎11-12-2025 06:34 AM

Hello,

I finally used he AppInsights agent from OpenTelemetry which is documented in the official Microsoft documentation here:

Below is an adaptation of this "Get started with OpenTelemetry" for an Azure Databricks compute.
1 - Upload the AppInsights.jar in the DBFS
2 - Make the jar to be uploaded in the compute.
From the settings of your compute, tab "configuration", then "advanced options", add an "init scripts" that will copy the jar on the compute.
This is what a really sample "init_script.sh" can looks like:
!/bin/bash
cp -f /dbfs/databricks/applicationinsights-agent-3.7.4.jar /databricks/jars
 
3 - Make the spark JVM to use the agent
This is achieved with a spark option that has to be applied to the driver and the executor (if you want to collect logs for both).
From the settings of your compute, tab "configuration", then "advanced options", add lines in "spark config".
spark.driver.extraJavaOptions -javaagent:/databricks/jars/applicationinsights-agent-3.7.4.jar
spark.executor.extraJavaOptions -javaagent:/databricks/jars/applicationinsights-agent-3.7.4.jar

4 - Configure the AppInsights agent to report to your AppInsights.
This is achieved with an environment variable (how to retrieve this connection string is already explained in the Microsoft link at the top of this post)
From the settings of your compute, tab "configuration", then "advanced options", add a line in "Spark : environment variables".
For example:
APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=XXXXXXXXXXXXXXX;IngestionEndpoint=https://XXXX.applicationinsights.azure.com/;LiveEndpoint=https://francecentral.livediagnostics.monit...

That's it, now your logs will be sent to the configured appInsights.
Of course, you can do advanced configuration for the AppInsights agent (agent role name, sampling etc..) by putting an applicationInsights.json file at the same location that the jar (adapt step 1 and 2 then)
More information on this topic here:

Regards,
Loïc
0 Kudos