Databricks Community

anil_reddaboina · 06-05-2025

Hi Team,Recently we migrated the spark jobs from self hosted spark(YARN) Cluster to Databricks.Currently we are using the Databricks workflows with Job_Compute clusters and the Job Type - Spark JAR type execution, so when we run the job in databricks...

anil_reddaboina · 02-24-2025

Currently we are facing a challenge with below use case:The Airflow DAG has 4 tasks (Task1, Task2, Task3 and Task4) and The dependency is like thisTask 1>> Task2 >> Task3 >> Task4 (All tasks are spark-jar task typesIn Airflow DAG for Task2, there is ...

anil_reddaboina · 06-06-2025

Hey Brahma,Thanks for your reply. As a first step I will disable AQE config and test it. We are using the node pools with job_compute cluster type so that its not spinning up a new cluster for each Job. I'm configuring the below two configs also, do ...

anil_reddaboina · 03-02-2025

Thanks for your messageCreating the "Run if"or condition task type is currently not available with Airflow databricks provider.

Databricks Community

User Stats

User Activity

Slow running Spark job issue - due to the unknown spark stages created by Databircks Compute cluster

Databricks tasks are not skipping if running tasks using Airflow DatabricksworkflowTaskgroup

Re: Slow running Spark job issue - due to the unknown spark stages created by Databircks Compute clu

Re: Databricks tasks are not skipping if running tasks using Airflow DatabricksworkflowTaskgroup