I have a databricks job having two task those will run each alone or both parallel (will be controlled by if conditional task). When it runs parallel, one task is running for long time, but the same task finish quick when it runs alone. particularly the same one task is running for long when runs both in parallel, futher analysis identified that the specific task (databricks task) is assigned with FIFO pool in spark ui, other one is FAIR.
What could be the cause of long running that one same task. Can we use Dynamic resource allocation in job cluster for this scenario or anything to do with schedular pool.
