topic Re: Working locally then moving to databricks in Data Engineering

Working locally then moving to databricks

GC-James — Tue, 01 Feb 2022 11:25:53 GMT

Hello DataBricks,

Struggling with a workflow issue and wondering if anyone can help. I am developing my project in R and sometimes Python locally on my laptop, and committing the files to a git repo. I can then clone that repo in databricks, and *see* the files, but I can't run them. Is there any reccomended workflow or way to do this? The picture below is an example of how my R script appears in Databricks once I have imported the repository. I can't attach to a cluster, and I can't run any of the lines/commands.

Thanks, James

Re: Working locally then moving to databricks

Hubert-Dudek — Tue, 01 Feb 2022 15:08:09 GMT

This is separate script which than need to be run from notebook (or job). I am not using R but in Python and Scala it works the same. In Python I am just importing it in notebook ("from folder_structure import myClass") in R probably similar. There are also other ways as well; for example develop in notebooks or use RStudio on databricks https://docs.databricks.com/spark/latest/sparkr/rstudio.html

Re: Working locally then moving to databricks

GC-James — Fri, 04 Feb 2022 21:02:01 GMT

Kind of thanks. It tells me how to 'work around' it. But I think it's a shame that a notebook can't recognise/parse a script to allow interactive running of commands, rather than having to run or source the whole script.