topic Re: Scheduling jobs with Airflow result in each task running multiple jobs. in Data Engineering

Scheduling jobs with Airflow result in each task running multiple jobs.

Tacuma — Wed, 11 Jan 2023 22:04:35 GMT

Hey everyone, I'm experiementing with running containerized pyspark jobs in Databricks, and orchestrating them with airflow. I am however, encountering an issue here. When I trigger an airflow DAG, and I look at the logs, I see that airflow is spinning up multiple jobs in its first try.

It's really strange. Attached a screenshot - in this instance, I need the job to run only once, but here 3 jobs are being run.

Wanted to know if this has been encountered before and any fixes, and will send cluster and pool config details upon request.

Re: Scheduling jobs with Airflow result in each task running multiple jobs.

daniel_sahal — Thu, 12 Jan 2023 06:57:28 GMT

That's weird behaviour. Can you please share the sample of an Airflow code?

Re: Scheduling jobs with Airflow result in each task running multiple jobs.

Tacuma — Fri, 13 Jan 2023 21:44:36 GMT

Thanks Daniel. Sure thing.

Re: Scheduling jobs with Airflow result in each task running multiple jobs.

Tacuma — Mon, 16 Jan 2023 12:51:01 GMT

Both, I guess? Yes, all jobs share the same config - the question I have is why in the same airflow task log, there are 3 jobs runs. I'm hoping that there's something in the configs and may give me some kind of clue.

Re: Scheduling jobs with Airflow result in each task running multiple jobs.

Debayan — Mon, 16 Jan 2023 04:39:29 GMT

Hi @Tacuma Solomon , 3 jobs with the same config? or 3 job runs?