Log jobs stdout to an Azure Logs Analytics workspace

[templier2]

Hello,

I have enabled cluster logs sending through an mspnp/spark-monitoring, but I don't see there stdout/stderr/log4j logs.

Is it supported?

[VZLA]

Hi @templier2 thanks for your question!

Have you tried enabling "Cluster log delivery" in Databricks, which can write these logs to an Azure storage account. Once there, I believe you can use a pipeline or Azure Log Analytics agent to forward these logs to your Log Analytics workspace.

[templier2]

Hello @VZLA , thank you for your answer.

I can't see there an option to put logs to a storage container - only to the DBFS mount point.

I can only export logs to a mount point on DBFS.
Mounting a storage container there is only possible via an init script, but altogether, it feels like a makeshift solution held together with duct tape and chewing gum.

[VZLA]

Hi @templier2 

If it works, it’s not duct tape and chewing gum; it’s a paperclip away from advanced engineering! 🙃

You're right, I forgot this option is only there for AWS/S3. So, yeah I think that's the current and only way, mount points.