topic Re: Apart from notebook , is it possible to deploy an application (Pyspark , or R+spark) as a package or file and execute them in Databricks ? in Data Engineering

Apart from notebook , is it possible to deploy an application (Pyspark , or R+spark) as a package or file and execute them in Databricks ?

sunil_smile — Tue, 23 Nov 2021 16:04:53 GMT

Hi,

With the help of Databricks-connect i was able to connect the cluster to my local IDE like Pycharm and Rstudio desktop version and able to develop the application and committed the code in Git.

When i try to add that repo to the Databricks workspace , i noticed that python files which i created in Pycharm are not getting displayed. I see only the notebooks file.

Is there any option , to deploy those python files in databricks cluster and execute those files.

files present in pycharm

but those files are not getting displayed in databricks workspace.

Most of my developers prefer to work in the IDE which they are familiar and comfortable. It helps them to develop and debug them quickly.

Kindly suggest , how i can address this problem.

Re: Apart from notebook , is it possible to deploy an application (Pyspark , or R+spark) as a package or file and execute them in Databricks ?

Hubert-Dudek — Tue, 23 Nov 2021 16:59:46 GMT

Please enable option "Files in Repo" in Settings -> Admin Console

Re: Apart from notebook , is it possible to deploy an application (Pyspark , or R+spark) as a package or file and execute them in Databricks ?

sunil_smile — Wed, 24 Nov 2021 08:51:53 GMT

@Hubert Dudek Thanks for your response.

Yes now i can able to see the python files.

But how i can execute that python file , because i see the Job accepts only DBFS file location..

Is there any other way to execute that python file?

Our requirement is to do the development in IDEs Pycharm or Rstudio using the Databricks-connect and deploy the final version of code in databricks and execute them as Job. Is there any option available in Databricks

Re: Apart from notebook , is it possible to deploy an application (Pyspark , or R+spark) as a package or file and execute them in Databricks ?

Hubert-Dudek — Wed, 24 Nov 2021 11:23:29 GMT

I just import class from .py in notebook and then use it:

from folder.file_folder.file_name import class

in notebook I initiate class.

I have everything in repos (notebook, py files). I have repo opened in Visual Studio and I can edit there notebooks as well. I haven't tested other IDE but it is just git repo with notebook so you can edit as well in PyCharm. Than in job you can set just notebook so it will be much easier.

Re: Apart from notebook , is it possible to deploy an application (Pyspark , or R+spark) as a package or file and execute them in Databricks ?

sunil_smile — Wed, 24 Nov 2021 16:06:42 GMT

So we definitely need a notebook just to execute our python file.

Is it possible to deploy this python files in the Workspace , instead of the Repos ?

Re: Apart from notebook , is it possible to deploy an application (Pyspark , or R+spark) as a package or file and execute them in Databricks ?

Atanu — Tue, 30 Nov 2021 06:52:16 GMT

may be you will be interested our db connect . not sure if that resolve your issue to connect with 3rd party tool and setup ur supported IDE notebook server

https://docs.databricks.com/dev-tools/databricks-connect.html