topic Python Logging cant save log in DBFS in Get Started Discussions

Python Logging cant save log in DBFS

Kaizen — Mon, 05 Feb 2024 23:06:48 GMT

Hi! I am trying to integrate logging into my project. Got the library and logs to work but cant log the file into DBFS directly.

Have any of you been able to save and append the log file directly to dbfs? From what i came across online the best way to do this is log it locally on your cluster and then copy or move the file to dbfs when done.

@DataBricks - it would be great to do this to monitor issues that might occur during the runtime

Re: Python Logging cant save log in DBFS

feiyun0112 — Tue, 06 Feb 2024 02:15:18 GMT

you can use azure_storage_logging

Set Python Logging to Azure Blob, but Can not Find Log File there - Stack Overflow

Re: Python Logging cant save log in DBFS

Kaizen — Tue, 06 Feb 2024 21:41:11 GMT

Nice I will be trying that shortly BUTTTT

Another way to do this is through enabling logging on the cluster on dbfs and then adding custom logging on top of the spark logging. These two articles walk through the steps:

1) https://community.databricks.com/t5/data-engineering/pyspark-logging-custom-to-azure-blob-mount-directory/m-p/11007#M6048

2) https://community.databricks.com/t5/data-engineering/how-to-add-i-custom-logging-in-databricks/m-p/20831