topic Re: Local Development on Databricks in Data Engineering

Local Development on Databricks

isaac_gritz — Tue, 23 Aug 2022 07:17:30 GMT

How to Develop Locally on Databricks with your Favorite IDE

dbx is a Databricks Labs project that allows you to develop code locally and then submit against Databricks interactive and job compute clusters from your favorite local IDE (AWS | Azure | GCP) such as VS Code, PyCharm, IntelliJ, or Eclipse.

dbx is an extension of the Databricks CLI and also makes it easy to manage multiple execution environments and deployment configurations as well as pre-built templates for integration with popular CI tools such as GitHub Actions, Azure DevOps, and GitLab.

Databricks has an official extension for VS Code to be able to execute code written locally against jobs or all purpose clusters. In addition, there is an official Databricks Driver for SQLTools in VS Code to browse SQL objects and run SQL queries in Databricks workspaces from VS Code.

Let us know in the comments if you have had a chance to test out dbx or our VS Code plugins for local IDE development!

Re: Local Development on Databricks

-werners- — Tue, 23 Aug 2022 07:32:04 GMT

I use it for local development of our libraries. Works fine, but I did not yet use it to submit to clusters.

Re: Local Development on Databricks

matiasm — Tue, 30 Aug 2022 14:58:33 GMT

I've found "dbx" really interesting. In particular it makes interactions with databricks from the local environment very smooth. I love how the dbx documentation describes the entire development process: it's the first time I see support for good engineering practices on the development phase.

There's one use case that dbx is not helping me with. I would like to develop a model locally using only pyspark and accessing the data on dbfs. I'm willing to use "dbutils", but to run it locally I need 'databricks-connect', which doesn't support databricks runtime 11, the latest one. Is there any other way to use dbutils locally?

Re: Local Development on Databricks

isaac_gritz — Wed, 31 Aug 2022 16:56:49 GMT

Hi @Matias Marenchino unfortunately you cannot run dbutils locally but if you can use dbx execute against an interactive cluster for a more interactive development experience.

Re: Local Development on Databricks

xiangzhu — Fri, 18 Nov 2022 14:51:07 GMT

dbx is great for deploy, but hopefully spark connect could be released as soon as possible

Re: Local Development on Databricks

slowder — Tue, 03 Jan 2023 14:51:07 GMT

We could build a helper function that detects if we are running on a generic pyspark versus on a Databricks cluster. That way, when databricks dbutils aren't available, we'd have a stand-in that would allow us to work disconnected until our code is ready to deploy to a cluster. @Isaac Gritz

Re: Local Development on Databricks

Jfoxyyc — Wed, 04 Jan 2023 21:04:53 GMT

I'm actually not a fan of dbx. I prefer the AWS Glue interactive sessions way of using the IDE. It's exactly like the web notebook experience. I can see the reason why dbx exists, but I'd still like to use a regular notebook experience in my IDE.

Re: Local Development on Databricks

kmodelew — Tue, 15 Apr 2025 13:10:59 GMT

Hi,

You can use any of existing IDE. I'm using pycharm. I have created my own utils to run code on databricks. In .env file I have environmental variables and using SDK I'm creating SparkSession object and WorkspaceObject that you can use to read credentials, etc. (gives you dbutils functionality). In your pyproject.toml you need only databricks connect library (version for your cluster runtime).

"databricks-connect == 15.4.6",

#ENV Variables for Databricks SDK
DATABRICKS_HOST =https://....
DATABRICKS_TOKEN=xxxxx
CLUSTER_ID=xxxx