topic Re: Unable to create log files using logging.basicConfig() in Data Engineering

Unable to create log files using logging.basicConfig()

HariPrasad1 — Thu, 19 Jun 2025 14:29:39 GMT

When I run this code below, I am not able to see the file under the path specified:

import logging

logger = logging.getLogger(__name__)

logging.basicConfig(filename='/Volumes/d_use1_ach_dbw_databricks1/default/ach_elegibility_raw/logs/example.log', encoding='utf-8', level=logging.DEBUG)

logger.debug('This message should go to the log file')

logger.info('So should this')

logger.warning('And this, too')

logger.error('And non-ASCII stuff, too, like Øresund and Malmö')

How do we locate the file in this scenario?

Re: Unable to create log files using logging.basicConfig()

Yogesh_Verma_ — Thu, 19 Jun 2025 16:44:57 GMT

The issue is happening because you're calling logging.getLogger(__name__) before setting up logging.basicConfig(). When the logger is created too early, it doesn't know about the file handler, so it doesn't write to the file.

To fix this, make sure you configure logging before creating or using the logger. Here's the corrected version:

------------------------------------
import logging
import os

# Make sure the log directory exists
os.makedirs('/Volumes/d_use1_ach_dbw_databricks1/default/ach_elegibility_raw/logs/', exist_ok=True)

# Configure logging first
logging.basicConfig(
filename='/Volumes/d_use1_ach_dbw_databricks1/default/ach_elegibility_raw/logs/example.log',
encoding='utf-8',
level=logging.DEBUG,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

# Now get the logger
logger = logging.getLogger(__name__)

# Log messages
logger.debug('This message should go to the log file')
logger.info('So should this')
logger.warning('And this, too')
logger.error('And non-ASCII stuff, too, like Øresund and Malmö')
Also, if you're running this inside Databricks, local file paths like /Volumes/... might not work. In that case, update the file path to:

------------------------------------------------
/dbfs/Volumes/d_use1_ach_dbw_databricks1/default/ach_elegibility_raw/logs/example.log
Then you can view the log file using:

----------------------------
%fs head /Volumes/d_use1_ach_dbw_databricks1/default/ach_elegibility_raw/logs/example.log

Re: Unable to create log files using logging.basicConfig()

HariPrasad1 — Fri, 20 Jun 2025 06:39:34 GMT

This is working Fine. But do you know how can we generate different log files for different runs?