Hi,

I am using the logger to log some parameters in my code and I want to save the file under DBFS. But for some reason the file is not getting created under DBFS. If I clear the state of the notebook and check the DBFS dir then file is present. Please let me know if anyone has any idea regarding this. I already tried all possible solutions available on internet but none of them fix the issue.

I have pasted the code below for reproducing the issue.

import logging

log_file = f"e.log"

logging.basicConfig(

  filename="/dbfs/FileStore/" + log_file,

  format="[%(filename)s:%(lineno)s %(asctime)s] %(message)s",

  level=logging.INFO,

  force=True

)

__LOG__ = logging.getLogger(__name__)

__LOG__.info("Starting ETL staging pipeline only!")

__LOG__.info("Starting ETL staging pipeline only!")

display(dbutils.fs.ls("dbfs:/FileStore/"))