Running a jar on Databricks shared cluster using Airflow

ayush19
New Contributor III

Hello,

I have a requirement to run a jar already installed on a Databricks cluster. It needs to be orchestrated using Apache Airflow. 

I followed the docs for the operator which can be used to do so https://airflow.apache.org/docs/apache-airflow-providers-databricks/1.0.0/operators.html 

The issue is, that every time I run this DAG, the cluster restarts and the jar file is installed on the cluster again. The file is already stored in Volume and installed on cluster, yet it restarts and re-installs jar on cluster. 

How can I avoid this?