TaskSensor - check if task is succeded

kmodelew — Tue, 26 Mar 2024 10:05:49 GMT

Hi,

I would like to check if the task within job is succeded (even the job is marked as failed because on of the tasks).

I need to create dependency for tasks within other jobs. The case is that I have one job for loading all tables for one country. Report uses some tables from job.

I'm looking something similar to ExternalTasSensor in Airflow.

What is the best practice in this area? I see the option to use API but maybe you see other, better possibilites.

Re: TaskSensor - check if task is succeded

kmodelew — Tue, 26 Mar 2024 14:00:00 GMT

@Retired_mod

Thank you. In my case I plan to use API:

https://docs.databricks.com/api/workspace/jobs/listruns - for choosing the latest run_id

https://docs.databricks.com/api/workspace/jobs/getrun - for getting task status for run_id

Python script will run every 10 minutes in certain predefined period of time and check the status of the tasks.

topic Re: TaskSensor - check if task is succeded in Data Engineering

TaskSensor - check if task is succeded

Re: TaskSensor - check if task is succeded