TaskSensor - check if task is succeded
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-26-2024 03:04 AM - edited 03-26-2024 03:05 AM
Hi,
I would like to check if the task within job is succeded (even the job is marked as failed because on of the tasks).
I need to create dependency for tasks within other jobs. The case is that I have one job for loading all tables for one country. Report uses some tables from job.
I'm looking something similar to ExternalTasSensor in Airflow.
What is the best practice in this area? I see the option to use API but maybe you see other, better possibilites.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-26-2024 06:58 AM - edited 03-26-2024 07:00 AM
Thank you. In my case I plan to use API:
https://docs.databricks.com/api/workspace/jobs/listruns - for choosing the latest run_id
https://docs.databricks.com/api/workspace/jobs/getrun - for getting task status for run_id
Python script will run every 10 minutes in certain predefined period of time and check the status of the tasks.