topic Re: logging.basicConfig not creating a file in Databricks in Data Engineering

logging.basicConfig not creating a file in Databricks

chandan_a_v — Thu, 09 Jun 2022 11:20:39 GMT

Hi,

I am using the logger to log some parameters in my code and I want to save the file under DBFS. But for some reason the file is not getting created under DBFS. If I clear the state of the notebook and check the DBFS dir then file is present. Please let me know if anyone has any idea regarding this. I already tried all possible solutions available on internet but none of them fix the issue.

I have pasted the code below for reproducing the issue.

import logging

log_file = f"e.log"

logging.basicConfig(

filename="/dbfs/FileStore/" + log_file,

format="[%(filename)s:%(lineno)s %(asctime)s] %(message)s",

level=logging.INFO,

force=True

)

__LOG__ = logging.getLogger(__name__)

__LOG__.info("Starting ETL staging pipeline only!")

display(dbutils.fs.ls("dbfs:/FileStore/"))

Re: logging.basicConfig not creating a file in Databricks

chandan_a_v — Thu, 09 Jun 2022 11:21:27 GMT

@Kaniz Fatma ,

Please let me know if you have any idea regarding this.

Re: logging.basicConfig not creating a file in Databricks

chandan_a_v — Fri, 10 Jun 2022 09:55:06 GMT

@Kaniz Fatma,

Did you find any solution? Please let me know.

Re: logging.basicConfig not creating a file in Databricks

Anonymous — Fri, 10 Jun 2022 10:26:02 GMT

Perhaps PyCharm sets a different working directory, meaning the file ends up in another place. Try providing a full path.

Re: logging.basicConfig not creating a file in Databricks

chandan_a_v — Fri, 10 Jun 2022 10:45:36 GMT

Hi @Halen15 Noyes ,

I am executing the notebook in Databricks. And there we save the file under DBFS.

Re: logging.basicConfig not creating a file in Databricks

chandan_a_v — Fri, 10 Jun 2022 19:56:28 GMT

Hi @Kaniz Fatma ,

I am using 10.4 LTS.

Re: logging.basicConfig not creating a file in Databricks

chandan_a_v — Tue, 14 Jun 2022 14:12:44 GMT

Hi @Kaniz Fatma,

As I mentioned in my problem description the file is not getting created only. But if I clear the state of the notebook and check the DBFS dir the file is present. As

I am creating the log file in the notebook and need to upload it to AWS s3 location it has to work how normal python env works. I would recommend using the code I posted to reproduce the issue and fix it from Databrick's side. As other people might also face the same issue in the future.

Re: logging.basicConfig not creating a file in Databricks

chandan_a_v — Fri, 17 Jun 2022 06:39:09 GMT

Hi @Kaniz Fatma,

Any update on my request? please fix this issue from your end ASAP.

Thanks,

Chandan

Re: logging.basicConfig not creating a file in Databricks

chandan_a_v — Wed, 13 Jul 2022 07:34:08 GMT

HI @Kaniz Fatma ,

The issue can be solved by replacing the file handler with a stream handler that contains the string io object. Also please reply to our question when you have some free time. And when someone points out a bug in Databricks it's good to notify the developing team.

Thanks,

Chandan