Scheduling jobs with Airflow result in each task running multiple jobs.

[Tacuma]

Hey everyone, I'm experiementing with running containerized pyspark jobs in Databricks, and orchestrating them with airflow. I am however, encountering an issue here. When I trigger an airflow DAG, and I look at the logs, I see that airflow is spinning up multiple jobs in its first try.

It's really strange. Attached a screenshot - in this instance, I need the job to run only once, but here 3 jobs are being run.

Wanted to know if this has been encountered before and any fixes, and will send cluster and pool config details upon request.

[daniel_sahal]

That's weird behaviour. Can you please share the sample of an Airflow code?

[Tacuma]

Thanks Daniel. Sure thing.

[Debayan]

Hi @Tacuma Solomon​ , 3 jobs with the same config? or 3 job runs?

[Tacuma]

Both, I guess? Yes, all jobs share the same config - the question I have is why in the same airflow task log, there are 3 jobs runs. I'm hoping that there's something in the configs and may give me some kind of clue.