Ingesting logs from Databricks (GCP) to Azure log Analytics

[talenik]

Hi everyone, I wanted to ask if there is any way through which we can ingest logs from GCP databricks to azure log analytics in store-sync fashion. 

Meaning we will save logs into some cloud bucket lets say, then from there we should be able to send logs to Azure log analytics asynchronous. This should not hamper productivity or speed of application emitting logs. 

Can you please help with this

Thanks in advance!!

[Kaniz_Fatma]

Hi @talenik, To ingest logs from Google Cloud Platform (GCP) Databricks into Azure Log Analytics in an asynchronous manner, you can follow these steps:

1. Configure Databricks Logs in JSON Format:

   • Set up your Databricks cluster to format logs in JSON.
   • This ensures that the logs are structured and can be easily processed.

2. Send Logs to Azure Log Analytics (ALA):

   • Use an HTTP POST request to send logs to the Azure Monitor HTTP Event Collector (HEC) endpoint.
   • Include the Workspace ID in the URL and the Primary Key as a custom header in your request.
   • This setup allows seamless ingestion of Databricks logs into Azure Log Analytics for monitoring and ...¹.

3. Consider Using the Log4j Appender:

   • If you’re using Azure Databricks, consider using the Log4j appender.
   • Build the spark-listeners-1.0-SNAPSHOT.jar and spark-listeners-loganalytics-1.0-SNAPSHOT.jar JAR files as described in the GitHub readme.
   • Create a log4j.properties configuration file for your application.
   • In your application code, include the spark-listeners-loganalytics project and import com.microsoft....².

If you encounter any issues, feel free to ask for further assistance! 😊

[talenik]

Hello @Kaniz_Fatma ,

Thanks for your help. Sending logs at real-time with workspace URL sounds good but it can add delay in your jobs execution since requests will be triggered sequentially.

We are looking something like spark-monitoring for GCP databricks, but spark-monitoring library is specially built for Azure databricks, will it work like this on GCP databricks?

GCP databricks is build different than Azure databricks and hence internal paths are different also.
I tried using it for GCP databricks, but since there is difference in resource hierarchy between gcp and azure I am unsure if this can be used. 

Also, we are using 14.3 LTS version and for now I am getting class not found exceptions while using the jar for scala classes.