Pyspark logging - custom to Azure blob mount directory

[kjoth]

I'm using the logging module to log the events from the job, but it seems the log is creating the file with only 1 lines. The consecutive log events are not being recorded. Is there any reference for custom logging in Databricks.

[Kaniz]

Hi @karthick J​ , Please let me know if this thread helps:-

https://community.databricks.com/s/question/0D53f00001GHVNHCA5/how-to-add-i-custom-logging-in-databr...

[Kaniz]

Hi @karthick J​ , When you create your cluster in databricks, there is a tab where you can specify the log directory (empty by default).

Logs are written on DBFS, so you just have to specify the directory you want.

 

[kjoth]

Thanks for the answer.

Instead of cluster logs, i want log only specific program run logs to a file with rotating.Is there a way to write the logs to mounted blob storage directory with append mode. I have read that in data-bricks file write doesn't support append mode.

[jose_gonzalez · Accepted solution]

hi @karthick J​ ,

If you want to create a custom logger, then you will need to use log4j to create your logger. The first post will show you how to do it. If you want to saved your captured events, then you will need to follow the second post that Kaniz has shared. You will need to parse your data when reading it back.

[kjoth]

Thank you for the answer

[Anonymous]

@karthick J​ - If Jose's answer helped solve the issue, would you be happy to mark their answer as best so that others can find the solution more easily?