Azure Databricks job and exception handling

lcalca95 — Tue, 20 Dec 2022 08:36:18 GMT

Hi,

I'm working on Azure Databricks and I created two jobs, one based on a python wheel and the other based on a notebook, with the same code. The code get data from Azure blob storage, process data with pyspark and send data to EventHub. The whole code is wrapped in a try / except, like this one:

# import libraries
from opencensus.ext.azure.log_exporter import AzureLogHandler
import logging
 
# add before def main()..
logger = logging.getLogger()
logger.addHandler(
    AzureLogHandler(
        connection_string = 'my-connection-string'
    )
)
 
if __name__ == '__main__':
    try:
        main() # main code: get data, process data, send data to eventhub
    except Exception as e:
        msg = str(getattr(e, 'message', repr(e)))
        logger.exception(msg)

I tried to generate an exception to test logging in App Insights for both the wheel based job and the notebook based job, and I noticed that even if they worked fine the first one (wheel job) return "failed" in the end, instead the second one (notebook job) return "succeded".

Does someone knows why the behavior changes in relation to the job "format"?

Thanks for the help!

topic Azure Databricks job and exception handling in Data Engineering

Azure Databricks job and exception handling