Running a jar on Databricks shared cluster using Airflow
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-24-2024 11:05 PM
Hello,
I have a requirement to run a jar already installed on a Databricks cluster. It needs to be orchestrated using Apache Airflow.
I followed the docs for the operator which can be used to do so https://airflow.apache.org/docs/apache-airflow-providers-databricks/1.0.0/operators.html
The issue is, that every time I run this DAG, the cluster restarts and the jar file is installed on the cluster again. The file is already stored in Volume and installed on cluster, yet it restarts and re-installs jar on cluster.
How can I avoid this?