topic Re: Pyspark logging - custom to Azure blob mount directory in Data Engineering

Pyspark logging - custom to Azure blob mount directory

kjoth — Thu, 11 Nov 2021 12:37:20 GMT

I'm using the logging module to log the events from the job, but it seems the log is creating the file with only 1 lines. The consecutive log events are not being recorded. Is there any reference for custom logging in Databricks.

Re: Pyspark logging - custom to Azure blob mount directory

jose_gonzalez — Fri, 12 Nov 2021 23:33:57 GMT

hi @karthick J ,

If you want to create a custom logger, then you will need to use log4j to create your logger. The first post will show you how to do it. If you want to saved your captured events, then you will need to follow the second post that Kaniz has shared. You will need to parse your data when reading it back.

Re: Pyspark logging - custom to Azure blob mount directory

kjoth — Sat, 13 Nov 2021 15:02:07 GMT

Thanks for the answer.

Instead of cluster logs, i want log only specific program run logs to a file with rotating.Is there a way to write the logs to mounted blob storage directory with append mode. I have read that in data-bricks file write doesn't support append mode.

Re: Pyspark logging - custom to Azure blob mount directory

kjoth — Sat, 13 Nov 2021 15:03:27 GMT

Thank you for the answer

Re: Pyspark logging - custom to Azure blob mount directory

Anonymous — Sat, 13 Nov 2021 19:39:35 GMT

@karthick J - If Jose's answer helped solve the issue, would you be happy to mark their answer as best so that others can find the solution more easily?