write both logging error Pyspark and Python exceptions

seefoods — Thu, 15 Jan 2026 10:16:39 GMT

Hello guyz,

Happy new year and best wishes for all of us. I am catching both Pyspark and Python exceptions but i want to write this logging error inside a delta table when i logging. Someone knows the best practise for this ?

Thanks

Cordially,

Re: write both logging error Pyspark and Python exceptions

szymon_dybczak — Thu, 15 Jan 2026 11:04:16 GMT

Hi @seefoods ,

You can write decorator function and use it to perform custom logging.

from functools import wraps from pyspark.sql import SparkSession from datetime import datetime import traceback spark = SparkSession.builder.getOrCreate() def logger(success_table: str, error_table: str): def decorator(func): @wraps(func) def wrapper(*args, **kwargs): try: result = func(*args, **kwargs) # Success log log_df = spark.createDataFrame([{ "function_name": func.__name__, "status": "SUCCESS", "timestamp": datetime.now() }]) log_df.write.format("delta").mode("append").saveAsTable(success_table) return result except Exception as e: # Error log error_df = spark.createDataFrame([{ "function_name": func.__name__, "status": "ERROR", "timestamp": datetime.now(), "error_message": str(e), "stack_trace": traceback.format_exc() }]) error_df.write.format("delta").mode("append").saveAsTable(error_table) raise # Optional: re-raise exception return wrapper return decorator #function definition @logger(success_table="LOG_SUCCESS", error_table="LOG_ERROR") def load_data(a, b): print(a/b) return "Done" #calling the function load_data(20, 10) #this will log the success message to LOG_SUCCESS load_data(10, 0) #this will log the exception message to LOG_ERROR

Check below article for more details:

Custom Logging to Delta Tables in Databricks: Reduce Code Redundancy | by Krishnanand A | Medium

Re: write both logging error Pyspark and Python exceptions

seefoods — Thu, 15 Jan 2026 11:13:56 GMT

Thanks a lot @szymon_dybczak

Re: write both logging error Pyspark and Python exceptions

szymon_dybczak — Thu, 15 Jan 2026 11:45:56 GMT

No problem @seefoods 🙂

topic Re: write both logging error Pyspark and Python exceptions in Data Engineering

write both logging error Pyspark and Python exceptions

Re: write both logging error Pyspark and Python exceptions

Re: write both logging error Pyspark and Python exceptions

Re: write both logging error Pyspark and Python exceptions