The issue is happening because you're calling logging.getLogger(__name__) before setting up logging.basicConfig(). When the logger is created too early, it doesn't know about the file handler, so it doesn't write to the file.

To fix this, make sure you configure logging before creating or using the logger. Here's the corrected version:

------------------------------------
import logging
import os

# Make sure the log directory exists
os.makedirs('/Volumes/d_use1_ach_dbw_databricks1/default/ach_elegibility_raw/logs/', exist_ok=True)

# Configure logging first
logging.basicConfig(
filename='/Volumes/d_use1_ach_dbw_databricks1/default/ach_elegibility_raw/logs/example.log',
encoding='utf-8',
level=logging.DEBUG,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

# Now get the logger
logger = logging.getLogger(__name__)

# Log messages
logger.debug('This message should go to the log file')
logger.info('So should this')
logger.warning('And this, too')
logger.error('And non-ASCII stuff, too, like Øresund and Malmö')
Also, if you're running this inside Databricks, local file paths like /Volumes/... might not work. In that case, update the file path to:

------------------------------------------------
/dbfs/Volumes/d_use1_ach_dbw_databricks1/default/ach_elegibility_raw/logs/example.log
Then you can view the log file using:

----------------------------
%fs head /Volumes/d_use1_ach_dbw_databricks1/default/ach_elegibility_raw/logs/example.log

This is working Fine. But do you know how can we generate different log files for different runs?