cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Apart from notebook , is it possible to deploy an application (Pyspark , or R+spark) as a package or file and execute them in Databricks ?

sunil_smile
Contributor

Hi,

With the help of Databricks-connect i was able to connect the cluster to my local IDE like Pycharm and Rstudio desktop version and able to develop the application and committed the code in Git.

When i try to add that repo to the Databricks workspace , i noticed that python files which i created in Pycharm are not getting displayed. I see only the notebooks file.

Is there any option , to deploy those python files in databricks cluster and execute those files.

files present in pycharm

imagebut those files are not getting displayed in databricks workspace.

image

Most of my developers prefer to work in the IDE which they are familiar and comfortable. It helps them to develop and debug them quickly.

Kindly suggest , how i can address this problem.

5 REPLIES 5

Hubert-Dudek
Esteemed Contributor III

Please enable option "Files in Repo" in Settings -> Admin Console

image.png

sunil_smile
Contributor

@Hubert Dudek​ Thanks for your response.

Yes now i can able to see the python files.

But how i can execute that python file , because i see the Job accepts only DBFS file location..

Is there any other way to execute that python file?

image 

Our requirement is to do the development in IDEs Pycharm or Rstudio using the Databricks-connect and deploy the final version of code in databricks and execute them as Job. Is there any option available in Databricks

Hubert-Dudek
Esteemed Contributor III

I just import class from .py in notebook and then use it:

from folder.file_folder.file_name import class

in notebook I initiate class.

I have everything in repos (notebook, py files). I have repo opened in Visual Studio and I can edit there notebooks as well. I haven't tested other IDE but it is just git repo with notebook so you can edit as well in PyCharm. Than in job you can set just notebook so it will be much easier.

sunil_smile
Contributor

So we definitely need a notebook just to execute our python file.

Is it possible to deploy this python files in the Workspace , instead of the Repos ?

image

Atanu
Esteemed Contributor
Esteemed Contributor

may be you will be interested our db connect . not sure if that resolve your issue to connect with 3rd party tool and setup ur supported IDE notebook server

https://docs.databricks.com/dev-tools/databricks-connect.html

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!