topic Re: Python callback functions fail to trigger in Data Engineering

Python callback functions fail to trigger

lozik — Mon, 22 Jul 2024 19:21:19 GMT

How can I get sys.exceptionhook and atexit module to trigger a callback function on exit of a python notebook? These fail to work when an unhandled exception is encountered (exceptionhook), or the program exits (atexit).

Re: Python callback functions fail to trigger

lozik — Mon, 29 Jul 2024 17:02:18 GMT

Hi @Retired_mod - `sys.excepthook` fails to trigger on an unhandled exception. Here's some example code using the logging module. This will write all logging messages to the logfile, except the the ones in `handleException`

import logging import sys import datetime import os logger = logging.getLogger("log_test") def handleException(excType, excValue, traceback, logger=logger): logger.info("message from exception handler") logger.error("Uncaught exception", exc_info=(excType, excValue, traceback)) sys.excepthook = handleException log_filepath = f'/dbfs/mnt/logs/log_test_{datetime.datetime.now().strftime("%Y%m%d%H%M%S")}.txt' logging.basicConfig( format=f'%(asctime)s - %(name)s - %(levelname)s - %(funcName)s - %(lineno)d - %(message)s', level=logging.INFO, handlers=[ logging.FileHandler(log_filepath), logging.StreamHandler(stream=sys.stdout) ] ) logger.debug("this is an debug message") logger.info("this is an info message") logger.warning("this is an warning message") logger.critical("this is an critical message") raise RuntimeError("Test unhandled")

Re: Python callback functions fail to trigger

Pieter — Mon, 17 Feb 2025 13:22:28 GMT

Hey Lozik,

Ran into this myself as well. The reason this doesn't work is because Databricks is using Ipython under the hood.

The following codesnippet creates an exception hook for all exceptions (using the general Exception), it's also possible to specify which exception you wish to add an hook. `exception_handler` was the function I used for handling the exceptions

# Add hook to exception handling ipy = get_ipython() ipy.set_custom_exc((Exception,), exception_handler)

Additional documentation can be found here: Document

Good luck!