using VSCode extension to interact with Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-29-2024 12:30 AM
Hello community,
I want to understand if it is possible to use Databricks Connect inside VSCode IDE to interact with Notebooks in local interactively like in Databricks Notebook, Is it possible? Because now I can only use the cluster and wait after the cluster executes the entire script, but If I want to use the Notebook I would to be much Interactive.
Thanks for your answers
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-29-2024 01:02 AM
@jeremy98 wrote:Hello community,
I want to understand if it is possible to use Databricks Connect inside VSCode IDE to interact with Notebooks in local interactively like in Databricks Notebook, Is it possible? Because now I can only use the cluster and wait after the cluster executes the entire script, but If I want to use the Notebook I would to be much Interactive Wellstar MyChartThanks for your answers
Yes, it is possible to use Databricks Connect inside VSCode IDE to interact with Notebooks interactively! With the Databricks extension for Visual Studio Code, you can run and debug notebook cells one at a time or all at once, and see their results directly in the VSCode UI. This setup allows you to have a more interactive experience similar to working directly in a Databricks Notebook.
To set this up, you'll need to install the Databricks extension for VSCode and ensure you have Databricks Connect properly configured. Once everything is set up, you can run and debug your notebook cells interactively, with the Spark code running on the remote cluster and the results being sent back to your local environment
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-29-2024 01:17 AM
Hi,
Thanks for your answer, actually I wrote a piece of code like:
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.getOrCreate()
But I still have this error:
ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf' (/Users/Library/Caches/pypoetry/virtualenvs/rnc-data-pipelines-Jgi6KLI_-py3.13/lib/python3.13/site-packages/pyspark/sql/udtf.py)
Do u know how to fix this? And For debugging the cells I need to execute the notebook in a proper way?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-29-2024 06:34 AM
Anyone, knows which could be the problem 😞