https://community.databricks.com/t5/data-engineering/what-s-the-best-way-to-develop-apache-spark-jobs-from-an-ide/m-p/25978#M18120 <P>A number of people like developing locally using an IDE and then deploying. What are the recommended ways to do that with Databricks jobs?</P> Mon, 07 Jun 2021 17:53:57 GMT Anonymous 2021-06-07T17:53:57Z https://community.databricks.com/t5/data-engineering/what-s-the-best-way-to-develop-apache-spark-jobs-from-an-ide/m-p/25978#M18120 <P>A number of people like developing locally using an IDE and then deploying. What are the recommended ways to do that with Databricks jobs?</P> Mon, 07 Jun 2021 17:53:57 GMT https://community.databricks.com/t5/data-engineering/what-s-the-best-way-to-develop-apache-spark-jobs-from-an-ide/m-p/25978#M18120 Anonymous 2021-06-07T17:53:57Z https://community.databricks.com/t5/data-engineering/what-s-the-best-way-to-develop-apache-spark-jobs-from-an-ide/m-p/25979#M18121 <P>The Databricks Runtime and Apache Spark use the same base API. One can create Spark jobs that run locally and have them run on Databricks with all available Databricks features.</P><P></P><P>It is required that one uses SparkSession.builder.getOrCreate() to create the SparkSession. The SparkSession is created in the Databricks environment and is treated as a singleton.</P><P></P><P>In addition, one can also test using databricks connect. Databricks connect replaces Apache Spark/Pyspark on your local machine and allows for your local machine to execute jobs on a Databricks cluster. <A href="https://docs.databricks.com/dev-tools/databricks-connect.html" target="test_blank">https://docs.databricks.com/dev-tools/databricks-connect.html</A></P> Mon, 07 Jun 2021 17:57:56 GMT https://community.databricks.com/t5/data-engineering/what-s-the-best-way-to-develop-apache-spark-jobs-from-an-ide/m-p/25979#M18121 Anonymous 2021-06-07T17:57:56Z