cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
isaac_gritz
by • Databricks Employee
  • 16147 Views
  • 6 replies
  • 6 kudos

Local Development on Databricks

How to Develop Locally on Databricks with your Favorite IDEdbx is a Databricks Labs project that allows you to develop code locally and then submit against Databricks interactive and job compute clusters from your favorite local IDE (AWS | Azure | GC...

Data Engineering
Reply
  • 16147 Views
  • 6 replies
  • 6 kudos
View Replies
Latest Reply
Jfoxyyc
Valued Contributor
  • 6 kudos

I'm actually not a fan of dbx. I prefer the AWS Glue interactive sessions way of using the IDE. It's exactly like the web notebook experience. I can see the reason why dbx exists, but I'd still like to use a regular notebook experience in my IDE.

  • 6 kudos
5 More Replies
BorislavBlagoev
by • Valued Contributor III
  • 27842 Views
  • 33 replies
  • 14 kudos

Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

Data Engineering
Reply
  • 27842 Views
  • 33 replies
  • 14 kudos
View Replies
Latest Reply
bhuvahh
New Contributor II
  • 14 kudos

I think plain python code will run with databricks connect (if it is a python program you are writing), and spark sql can be done by spark.sql(...).

  • 14 kudos
32 More Replies
avnerrhh
by • New Contributor III
  • 4745 Views
  • 6 replies
  • 4 kudos

Resolved! How do I import class/functions so it work in Databricks and in my IDE

I already saw this postI want my code to work on both platforms (Databricks and PyCharm), is there any way to do it?

Data Engineering
Reply
  • 4745 Views
  • 6 replies
  • 4 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

yes.one way is to develop everything locally on your pc, so you also need to have spark installed.This is of course not ideal as you will not have some interesting stuff that databricks provides.But it can be done. What you have to do is create a whl...

  • 4 kudos
5 More Replies
William_Scardua
by • Valued Contributor
  • 9141 Views
  • 6 replies
  • 3 kudos

Resolved! How do you create a Sandbox in your data environment ?

Hi guys,How do you create a Sandbox in your data environment ? have any idea ?Azzure/AWS + Data Lake + Databricks

Data Engineering
Reply
  • 9141 Views
  • 6 replies
  • 3 kudos
View Replies
Latest Reply
missyT
New Contributor III
  • 3 kudos

In a sandbox environment, you will find the Designer enabled. You can activate Designer by selecting the design icon Designer. on a page, or by choosing the Design menu item in the Settings Settings menu.

  • 3 kudos
5 More Replies
kmartin62
by • New Contributor III
  • 5591 Views
  • 9 replies
  • 4 kudos

Resolved! Configure Databricks (spark) context from PyCharm

Hello. I'm trying to connect to Databricks from my IDE (PyCharm) and then run delta table queries from there. However, the cluster I'm trying to access has to give me permission. In this case, I'd go to my cluster, run the cell which gives me permiss...

Data Engineering
Reply
  • 5591 Views
  • 9 replies
  • 4 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

"I'm trying to connect to Databricks from my IDE (PyCharm) and then run delta table queries from there."If you are going to deploy later your code to databricks the only solutions which I see is to use databricks-connect or just make development envi...

  • 4 kudos
8 More Replies
dimoobraznii
by • New Contributor III
  • 6684 Views
  • 2 replies
  • 9 kudos

databricks-connect' is not recognized as an internal or external command, operable program or batch file on windows

Hello,I've installed databricks-connect on Windows 10:C:\Users\danoshin>pip install -U "databricks-connect==9.1.*" Collecting databricks-connect==9.1.* Downloading databricks-connect-9.1.2.tar.gz (254.6 MB) |████████████████████████████████| 2...

Data Engineering
Reply
  • 6684 Views
  • 2 replies
  • 9 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 9 kudos

@Dmitry Anoshin​ , that seems messed up.the best you can do is to remove databricks connect and also to uninstall any pyspark installation.And then follow the installation guide.It should work after following the procedure.I use a Linux VM for this p...

  • 9 kudos
1 More Replies
Anonymous
by • Not applicable
  • 1336 Views
  • 1 replies
  • 1 kudos

What's the best way to develop Apache Spark Jobs from an IDE (such as IntelliJ/Pycharm)?

A number of people like developing locally using an IDE and then deploying. What are the recommended ways to do that with Databricks jobs?

Data Engineering
Reply
  • 1336 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

The Databricks Runtime and Apache Spark use the same base API. One can create Spark jobs that run locally and have them run on Databricks with all available Databricks features.It is required that one uses SparkSession.builder.getOrCreate() to create...

  • 1 kudos
Labels