Python Logging cant save log in DBFS

[Kaizen]

Hi! I am trying to integrate logging into my project. Got the library and logs to work but cant log the file into DBFS directly.

Have any of you been able to save and append the log file directly to dbfs? From what i came across online the best way to do this is log it locally on your cluster and then copy or move the file to dbfs when done.

@DataBricks - it would be great to do this to monitor issues that might occur during the runtime

[feiyun0112]

you can use  azure_storage_logging

Set Python Logging to Azure Blob, but Can not Find Log File there - Stack Overflow

[Kaizen]

Nice I will be trying that shortly BUTTTT

Another way to do this is through enabling logging on the cluster on dbfs and then adding custom logging on top of the spark logging. These two articles walk through the steps:

1)  https://community.databricks.com/t5/data-engineering/pyspark-logging-custom-to-azure-blob-mount-dire...

2) https://community.databricks.com/t5/data-engineering/how-to-add-i-custom-logging-in-databricks/m-...