add custom logs and save in a folder logs

ramyav7796
New Contributor II

‎05-08-2025 12:00 AM

Hi,

I am trying to add custom logging functionality for my code. Please refer to the code I am using, I am trying to save my log files by creating a logs folder in my users workspace. 

My intent is to store dynamic custom log files each time I run my notebook.

 
cell1
def delete_existing_folders(path,logger😞
    logger.info('------> START - STEP1: If path exists delere the folder <------ \n')
    if os.path.exists(path):
        folders = [folder for folder in os.listdir(path) if os.path.isdir(os.path.join(path, folder))]
        if not folders:
            logger.info("No folder found")
        else:
            for folder in folders:
                folder_path = os.path.join(path, folder)
                shutil.rmtree(folder_path)
                logger.info(f"Deleted existing folder: {folder_path}")
    logger.info('------> END STEP1 <------ \n')
 

cell2

import logging
from datetime import datetime
import os

# Define the log directory
log_dir = "/Workspace/Users/ramya.v@point32health.org/CD/"

# Create a timestamp for the logger file
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
log_filename = os.path.join(log_dir, f'notebook_log_{timestamp}.log')

# Configure the logging
logging.basicConfig(
    filename=log_filename,
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

# Create a logger object
logger = logging.getLogger()
 
input_files_path = "/Volumes/abc/calls_data/"
 
delete_existing_folders(input_files_path,logger)
 
I am unable to create log files with this code, tried multiple ways but log files are not getting generated in either the users workspace nor in volumes.
 
Please suggest a way where I can implement custom logs for my code
 
0 Kudos

Louis_Frolio
Databricks Employee
Databricks Employee

‎05-08-2025 07:23 AM

Here are some suggestions for your consideration.
 
The issue with your custom logging setup seems to stem from attempting to save the log files in a path under "/Workspace/Users/ramya.v@point32health.org/CD/", which is not directly writable by your code in Databricks. Databricks workspaces utilize DBFS (Databricks File System), and regular filesystem paths like /Workspace operate differently within this environment.
To solve this issue and ensure that dynamic custom log files are correctly created and stored with each notebook run, you should:
  1. Use DBFS for the Log Path: Update the log_dir variable to use a directory within DBFS. For instance: python log_dir = "/dbfs/Workspace/Users/your.email@databricks.com/logs/" Replace "your.email@databricks.com" with your actual workspace email.
  2. Ensure the Folder Exists: Before attempting to create log files, ensure the directory exists on DBFS. You can create the folder programmatically: python import os log_dir = "/dbfs/Workspace/Users/your.email@databricks.com/logs/" if not os.path.exists(log_dir): os.makedirs(log_dir)
  3. Update Logging Configuration: Modify your logging configuration to use DBFS for storing logs: ```python from datetime import datetime import logging import os
    # Define the log directory and create if necessary log_dir = "/dbfs/Workspace/Users/louis.frolio@databricks.com/logs/" if not os.path.exists(log_dir): os.makedirs(log_dir)
    # Create a timestamp for the log file timestamp = datetime.now().strftime('%Y%m%d_%H%M%S') log_filename = os.path.join(log_dir, f'notebooklog{timestamp}.log')
    # Configure the logging logging.basicConfig( filename=log_filename, level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s' )
    # Create a logger object logger = logging.getLogger() logger.info("Logging has been initialized successfully.") ```
  4. Run and Verify: After making these changes, run the notebook. Verify the logs by navigating to the /Workspace/Users/louis.frolio@databricks.com/logs/ path in Databricks.
  5. DBFS Path Accessibility: After running the notebook, you can access and download the logs through the Databricks UI or use dbutils to manage the files. For example: python files = dbutils.fs.ls("dbfs:/Workspace/Users/louis.frolio@databricks.com/logs/") display(files)
By following these steps, the custom log files will be correctly generated and stored dynamically in a location accessible for further inspection. This approach ensures compatibility with Databricks' storage environment.
 
Cheers, Lou.
0 Kudos