Orchestrate Databricks jobs with Apache Airflow

Sourav-Kundu — Tue, 29 Oct 2024 12:02:37 GMT

You can Orchestrate Databricks jobs with Apache Airflow

The Databricks provider implements the below operators:

DatabricksCreateJobsOperator : Create a new Databricks job or reset an existing job

DatabricksRunNowOperator : Runs an existing Spark job run to Databricks using the api/2.1/jobs/run-now API endpoint.

DatabricksSubmitRunOperator : Submits a Spark job run to Databricks using the api/2.1/jobs/runs/submit API endpoint.

DatabricksCopyIntoOperator : Executes COPY INTO command in a Databricks SQL endpoint or a Databricks cluster.

DatabricksSqlOperator : Executes SQL code in a Databricks SQL endpoint or a Databricks cluster

https://docs.databricks.com/en/jobs/how-to/use-airflow-with-jobs.html

@Advika_

Re: Orchestrate Databricks jobs with Apache Airflow

Advika_ — Tue, 29 Oct 2024 14:21:04 GMT

Good one @Sourav-Kundu! Your clear explanations of the operators really simplify job management, plus the resource link you included makes it easy for everyone to dive deeper 👍.

topic Re: Orchestrate Databricks jobs with Apache Airflow in Community Articles

Orchestrate Databricks jobs with Apache Airflow

Re: Orchestrate Databricks jobs with Apache Airflow