topic Re: using VSCode extension to interact with Databricks in Data Engineering

using VSCode extension to interact with Databricks

jeremy98 — Fri, 29 Nov 2024 08:30:29 GMT

Hello community,
I want to understand if it is possible to use Databricks Connect inside VSCode IDE to interact with Notebooks in local interactively like in Databricks Notebook, Is it possible? Because now I can only use the cluster and wait after the cluster executes the entire script, but If I want to use the Notebook I would to be much Interactive.

Thanks for your answers

Re: using VSCode extension to interact with Databricks

carolgrey98 — Fri, 29 Nov 2024 09:02:51 GMT

@jeremy98 wrote:
Hello community,
I want to understand if it is possible to use Databricks Connect inside VSCode IDE to interact with Notebooks in local interactively like in Databricks Notebook, Is it possible? Because now I can only use the cluster and wait after the cluster executes the entire script, but If I want to use the Notebook I would to be much Interactive Wellstar MyChart
Thanks for your answers

Yes, it is possible to use Databricks Connect inside VSCode IDE to interact with Notebooks interactively! With the Databricks extension for Visual Studio Code, you can run and debug notebook cells one at a time or all at once, and see their results directly in the VSCode UI. This setup allows you to have a more interactive experience similar to working directly in a Databricks Notebook.

To set this up, you'll need to install the Databricks extension for VSCode and ensure you have Databricks Connect properly configured. Once everything is set up, you can run and debug your notebook cells interactively, with the Spark code running on the remote cluster and the results being sent back to your local environment

Re: using VSCode extension to interact with Databricks

jeremy98 — Fri, 29 Nov 2024 09:17:55 GMT

Hi,
Thanks for your answer, actually I wrote a piece of code like:

from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate()

But I still have this error:

ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf' (/Users/Library/Caches/pypoetry/virtualenvs/rnc-data-pipelines-Jgi6KLI_-py3.13/lib/python3.13/site-packages/pyspark/sql/udtf.py)

Do u know how to fix this? And For debugging the cells I need to execute the notebook in a proper way?

Re: using VSCode extension to interact with Databricks

jeremy98 — Fri, 29 Nov 2024 14:34:41 GMT

Anyone, knows which could be the problem 😞