add custom logs and save in a folder logs

ramyav7796 — Thu, 08 May 2025 07:00:48 GMT

Hi,

I am trying to add custom logging functionality for my code. Please refer to the code I am using, I am trying to save my log files by creating a logs folder in my users workspace.

My intent is to store dynamic custom log files each time I run my notebook.

cell1

def delete_existing_folders(path,logger😞

logger.info('------> START - STEP1: If path exists delere the folder <------ \n')

if os.path.exists(path):

folders = [folder for folder in os.listdir(path) if os.path.isdir(os.path.join(path, folder))]

if not folders:

logger.info("No folder found")

else:

for folder in folders:

folder_path = os.path.join(path, folder)

shutil.rmtree(folder_path)

logger.info(f"Deleted existing folder: {folder_path}")

logger.info('------> END STEP1 <------ \n')

cell2

import logging

from datetime import datetime

import os

# Define the log directory

log_dir = "/Workspace/Users/ramya.v@point32health.org/CD/"

# Create a timestamp for the logger file

timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')

log_filename = os.path.join(log_dir, f'notebook_log_{timestamp}.log')

# Configure the logging

logging.basicConfig(

filename=log_filename,

level=logging.INFO,

format='%(asctime)s - %(levelname)s - %(message)s'

)

# Create a logger object

logger = logging.getLogger()

input_files_path = "/Volumes/abc/calls_data/"

delete_existing_folders(input_files_path,logger)

I am unable to create log files with this code, tried multiple ways but log files are not getting generated in either the users workspace nor in volumes.

Please suggest a way where I can implement custom logs for my code

Re: add custom logs and save in a folder logs

Louis_Frolio — Thu, 08 May 2025 14:23:32 GMT

Here are some suggestions for your consideration.

The issue with your custom logging setup seems to stem from attempting to save the log files in a path under "/Workspace/Users/ramya.v@point32health.org/CD/", which is not directly writable by your code in Databricks. Databricks workspaces utilize DBFS (Databricks File System), and regular filesystem paths like /Workspace operate differently within this environment.

To solve this issue and ensure that dynamic custom log files are correctly created and stored with each notebook run, you should:

Use DBFS for the Log Path: Update the log_dir variable to use a directory within DBFS. For instance: python log_dir = "/dbfs/Workspace/Users/your.email@databricks.com/logs/" Replace "your.email@databricks.com" with your actual workspace email.
Ensure the Folder Exists: Before attempting to create log files, ensure the directory exists on DBFS. You can create the folder programmatically: python import os log_dir = "/dbfs/Workspace/Users/your.email@databricks.com/logs/" if not os.path.exists(log_dir): os.makedirs(log_dir)
Update Logging Configuration: Modify your logging configuration to use DBFS for storing logs: ```python from datetime import datetime import logging import os

# Define the log directory and create if necessary log_dir = "/dbfs/Workspace/Users/louis.frolio@databricks.com/logs/" if not os.path.exists(log_dir): os.makedirs(log_dir)

# Create a timestamp for the log file timestamp = datetime.now().strftime('%Y%m%d_%H%M%S') log_filename = os.path.join(log_dir, f'notebooklog{timestamp}.log')

# Configure the logging logging.basicConfig( filename=log_filename, level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s' )

# Create a logger object logger = logging.getLogger() logger.info("Logging has been initialized successfully.") ```
Run and Verify: After making these changes, run the notebook. Verify the logs by navigating to the /Workspace/Users/louis.frolio@databricks.com/logs/ path in Databricks.
DBFS Path Accessibility: After running the notebook, you can access and download the logs through the Databricks UI or use dbutils to manage the files. For example: python files = dbutils.fs.ls("dbfs:/Workspace/Users/louis.frolio@databricks.com/logs/") display(files)

By following these steps, the custom log files will be correctly generated and stored dynamically in a location accessible for further inspection. This approach ensures compatibility with Databricks' storage environment.

Cheers, Lou.

topic Re: add custom logs and save in a folder logs in Data Engineering

add custom logs and save in a folder logs

Re: add custom logs and save in a folder logs