cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Working locally then moving to databricks

GC-James
Contributor II

Hello DataBricks,

Struggling with a workflow issue and wondering if anyone can help. I am developing my project in R and sometimes Python locally on my laptop, and committing the files to a git repo. I can then clone that repo in databricks, and *see* the files, but I can't run them. Is there any reccomended workflow or way to do this? The picture below is an example of how my R script appears in Databricks once I have imported the repository. I can't attach to a cluster, and I can't run any of the lines/commands.

Thanks, James

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

This is separate script which than need to be run from notebook (or job). I am not using R but in Python and Scala it works the same. In Python I am just importing it in notebook ("from folder_structure import myClass") in R probably similar. There are also other ways as well; for example develop in notebooks or use RStudio on databricks https://docs.databricks.com/spark/latest/sparkr/rstudio.html

View solution in original post

2 REPLIES 2

Hubert-Dudek
Esteemed Contributor III

This is separate script which than need to be run from notebook (or job). I am not using R but in Python and Scala it works the same. In Python I am just importing it in notebook ("from folder_structure import myClass") in R probably similar. There are also other ways as well; for example develop in notebooks or use RStudio on databricks https://docs.databricks.com/spark/latest/sparkr/rstudio.html

Kind of thanks. It tells me how to 'work around' it. But I think it's a shame that a notebook can't recognise/parse a script to allow interactive running of commands, rather than having to run or source the whole script.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group