Job tasks monitoring

maikel — Wed, 13 May 2026 16:25:59 GMT

Hello Community,

We have a case in our project that we would like to solve in an elegant and scalable manner. As always, I would really appreciate your suggestions and experience.

In short:

We have a multi-step job consisting of 4 stages. In one of the stages, the work is split into tasks per business unit. For simplicity, let’s assume there are 500 tasks that must be completed before moving to the next step, where the final data save operation is performed.

For debugging and audit purposes, all tasks are stored in a Delta table with statuses such as IN PROGRESS, READY, COMPLETED, etc. Along with the status, we also store uniqueID and runId so we can identify which tasks belong to a specific job run.

The entire workflow is triggered from our backend API using the Databricks SDK. At the end, we use webhooks to notify whether the job completed successfully or failed.

What we would like to achieve is displaying task progress to the user, for example:

200/500 completed → 40%
250/500 completed → 50%
etc.

Since we already maintain a Delta table with task statuses, what would be the best way to communicate this progress back to our backend/frontend layer?

Are there any native Databricks mechanisms or recommended patterns for this kind of monitoring/progress reporting?

One important requirement is that we do not want to calculate or expose progress directly within the processing job itself. We would prefer either:

a separate monitoring job/process, or
an existing/native Databricks solution if available.

Thanks a lot in advance for your help and recommendations!

topic Job tasks monitoring in Data Engineering

Job tasks monitoring