topic Re: Error Logging and Orchastration In Databricks in Data Engineering

Error Logging and Orchastration In Databricks

Pratikmsbsvm — Wed, 30 Jul 2025 09:29:54 GMT

Hello,

May someone please Help me designing the Error Logging and How to do orchestration for Pipeline below.

I am pulling data from Bronze layer and pushing it to silver layer after transformation.

1. How to do Error Logging and where to store

2. How to orchestrate this pipeline. Can we do with Databricks itself.

Thanks a lot for your kind help

Re: Error Logging and Orchastration In Databricks

Pratikmsbsvm — Wed, 30 Jul 2025 12:00:59 GMT

Actually .

have to design logging in Databricks CI Satellite EDLAP.

Can I do it in ADLS Gen2 Silver layer or do I need to have any other component.

Can Someone please help me how we can have folder structure and can we have delta table for logging. what all parameter we can log and how it's value can be capture.

Please help me.

Thanks a lot

Re: Error Logging and Orchastration In Databricks

Brahmareddy — Wed, 30 Jul 2025 13:31:31 GMT

Hi Pratikmsbsvm,

How are you doing today? Great question, For error logging in your Bronze to Silver pipeline, yes, you can absolutely store logs in a Delta table, ideally in your Silver layer on ADLS Gen2. A good approach is to create a separate Delta table like error_logs where you capture useful details such as: timestamp, table name, pipeline step, error message, source file, and maybe a JSON column to store the problematic row if possible. Use try-except blocks in your PySpark or notebooks and append errors into this log table. As for orchestration, Databricks Workflows is a solid built-in option—you can schedule, chain tasks, and set up alerts or retries. You don’t need an extra tool unless your org requires it. Keep a clean folder structure, like /logs/errors/, and organize logs by date or pipeline. This setup will keep your pipeline more transparent and easier to monitor.

Regards,

Brahma

Re: Error Logging and Orchastration In Databricks

Pratikmsbsvm — Wed, 30 Jul 2025 13:48:36 GMT

@Brahmareddy : Thanks a lot. do you have any page which shows real implementation, if handy. Kindly share.