Ingesting logs from Databricks (GCP) to Azure log Analytics

[talenik]

Hi everyone, I wanted to ask if there is any way through which we can ingest logs from GCP databricks to azure log analytics in store-sync fashion. 

Meaning we will save logs into some cloud bucket lets say, then from there we should be able to send logs to Azure log analytics asynchronous. This should not hamper productivity or speed of application emitting logs. 

Can you please help with this

Thanks in advance!!

[talenik]

Hello @Retired_mod ,

Thanks for your help. Sending logs at real-time with workspace URL sounds good but it can add delay in your jobs execution since requests will be triggered sequentially.

We are looking something like spark-monitoring for GCP databricks, but spark-monitoring library is specially built for Azure databricks, will it work like this on GCP databricks?

GCP databricks is build different than Azure databricks and hence internal paths are different also.
I tried using it for GCP databricks, but since there is difference in resource hierarchy between gcp and azure I am unsure if this can be used. 

Also, we are using 14.3 LTS version and for now I am getting class not found exceptions while using the jar for scala classes. 

[talenik · Accepted solution]

Hi @Retired_mod ,

Thanks for help. We decided to develop our own library for logging to azure log analytics. We used buffer for this. 

We are currently on timer based logs but in future versions we wanted to move to memory based.

Thanks,

Nikhil