add custom logs and save in a folder logs

[ramyav7796]

Hi,

I am trying to add custom logging functionality for my code. Please refer to the code I am using, I am trying to save my log files by creating a logs folder in my users workspace. 

My intent is to store dynamic custom log files each time I run my notebook.

cell1

def delete_existing_folders(path,logger😞

    logger.info('------> START - STEP1: If path exists delere the folder <------ \n')

    if os.path.exists(path):

        folders = [folder for folder in os.listdir(path) if os.path.isdir(os.path.join(path, folder))]

        if not folders:

            logger.info("No folder found")

        else:

            for folder in folders:

                folder_path = os.path.join(path, folder)

                shutil.rmtree(folder_path)

                logger.info(f"Deleted existing folder: {folder_path}")

    logger.info('------> END STEP1 <------ \n')

cell2

import logging

from datetime import datetime

import os

# Define the log directory

log_dir = "/Workspace/Users/ramya.v@point32health.org/CD/"

# Create a timestamp for the logger file

timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')

log_filename = os.path.join(log_dir, f'notebook_log_{timestamp}.log')

# Configure the logging

logging.basicConfig(

    filename=log_filename,

    level=logging.INFO,

    format='%(asctime)s - %(levelname)s - %(message)s'

)

# Create a logger object

logger = logging.getLogger()

input_files_path = "/Volumes/abc/calls_data/"

delete_existing_folders(input_files_path,logger)

I am unable to create log files with this code, tried multiple ways but log files are not getting generated in either the users workspace nor in volumes.

Please suggest a way where I can implement custom logs for my code

[BigRoux]

Here are some suggestions for your consideration.

The issue with your custom logging setup seems to stem from attempting to save the log files in a path under "/Workspace/Users/ramya.v@point32health.org/CD/", which is not directly writable by your code in Databricks. Databricks workspaces utilize DBFS (Databricks File System), and regular filesystem paths like /Workspace operate differently within this environment.

To solve this issue and ensure that dynamic custom log files are correctly created and stored with each notebook run, you should:

1. Use DBFS for the Log Path: Update the log_dir variable to use a directory within DBFS. For instance: python log_dir = "/dbfs/Workspace/Users/your.email@databricks.com/logs/" Replace "your.email@databricks.com" with your actual workspace email.
2. Ensure the Folder Exists: Before attempting to create log files, ensure the directory exists on DBFS. You can create the folder programmatically: python import os log_dir = "/dbfs/Workspace/Users/your.email@databricks.com/logs/" if not os.path.exists(log_dir): os.makedirs(log_dir)
3. Update Logging Configuration: Modify your logging configuration to use DBFS for storing logs: ```python from datetime import datetime import logging import os
   # Define the log directory and create if necessary log_dir = "/dbfs/Workspace/Users/louis.frolio@databricks.com/logs/" if not os.path.exists(log_dir): os.makedirs(log_dir)
   # Create a timestamp for the log file timestamp = datetime.now().strftime('%Y%m%d_%H%M%S') log_filename = os.path.join(log_dir, f'notebooklog{timestamp}.log')
   # Configure the logging logging.basicConfig( filename=log_filename, level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s' )
   # Create a logger object logger = logging.getLogger() logger.info("Logging has been initialized successfully.") ```
4. Run and Verify: After making these changes, run the notebook. Verify the logs by navigating to the /Workspace/Users/louis.frolio@databricks.com/logs/ path in Databricks.
5. DBFS Path Accessibility: After running the notebook, you can access and download the logs through the Databricks UI or use dbutils to manage the files. For example: python files = dbutils.fs.ls("dbfs:/Workspace/Users/louis.frolio@databricks.com/logs/") display(files)

By following these steps, the custom log files will be correctly generated and stored dynamically in a location accessible for further inspection. This approach ensures compatibility with Databricks' storage environment.

Cheers, Lou.