topic Re: Log jobs stdout to an Azure Logs Analytics workspace in Data Engineering

Log jobs stdout to an Azure Logs Analytics workspace

templier2 — Mon, 09 Dec 2024 16:15:50 GMT

Hello,

I have enabled cluster logs sending through an mspnp/spark-monitoring, but I don't see there stdout/stderr/log4j logs.

Is it supported?

Re: Log jobs stdout to an Azure Logs Analytics workspace

VZLA — Mon, 09 Dec 2024 16:50:28 GMT

Hi @templier2 thanks for your question!

Have you tried enabling "Cluster log delivery" in Databricks, which can write these logs to an Azure storage account. Once there, I believe you can use a pipeline or Azure Log Analytics agent to forward these logs to your Log Analytics workspace.

Re: Log jobs stdout to an Azure Logs Analytics workspace

templier2 — Wed, 11 Dec 2024 16:54:26 GMT

Hello @VZLA , thank you for your answer.

I can't see there an option to put logs to a storage container - only to the DBFS mount point.

I can only export logs to a mount point on DBFS.
Mounting a storage container there is only possible via an init script, but altogether, it feels like a makeshift solution held together with duct tape and chewing gum.

Re: Log jobs stdout to an Azure Logs Analytics workspace

VZLA — Wed, 11 Dec 2024 17:58:27 GMT

Hi @templier2

If it works, it’s not duct tape and chewing gum; it’s a paperclip away from advanced engineering! 🙃

You're right, I forgot this option is only there for AWS/S3. So, yeah I think that's the current and only way, mount points.