- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-20-2024 03:05 AM - edited โ12-20-2024 03:06 AM
Hi Community,
Iโve encountered an issue with the Databricks Extension for VS Code that seems to contradict the documentation. According to the Databricks documentation, the extension supports magic commands like %sql when used with Databricks Connect:
However, in practice, this functionality doesnโt appear to work as expected.
Hereโs an example of the code I ran:
# Databricks notebook source
df = spark.sql("SELECT 1 AS from__python_cell")
df.show()
# COMMAND ----------
# MAGIC %sql
# MAGIC SELECT 1 AS from_sql_cell
The first Python cell executes correctly, displaying the expected result. However, the %sql magic command produces an error:
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1734690770.134581 5908 init.cc:229] grpc_wait_for_shutdown_with_timeout() timed out.
This raises several questions:
- Is %sql truly supported in the VS Code extension when using Databricks Connect, or is there a misunderstanding in the documentation?
- Could this be a version compatibility issue between Databricks Connect, the extension, and the runtime?
- Are there any workarounds to enable the use of %sql in this setup, or should we rely exclusively on spark.sql() in Python cells?
My setup includes:
- Databricks Connect: 13.3.2
- Databricks Runtime: 16.0
- Databricks Extension for VS Code: v2.5.0
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-28-2024 08:50 AM
are you trying to see it from the interactive screen as following screenshot:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-20-2024 04:03 AM
According to the documentation, the Databricks extension for Visual Studio Code supports the %sql magic command.
However, there are limitations to this support:
- The notebooks magics %r and %scala are not supported and will display an error if called.
- The notebook magic %sql does not support some DML commands, such as Show Tables.
Given your setup (Databricks Connect: 13.3.2, Databricks Runtime: 16.0, Databricks Extension for VS Code: v2.5.0), it appears that the %sql magic command should be supported.
Do you have option to open support case so this can be reviewed further and possibly engage engineering if a bug is detected?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-20-2024 04:17 AM
Hi @Walter_C ,
Thanks for the answer!
Just to update you I have also adjusted the versions so that all versions match:
- Databricks Runtime 15.4
- Databricks-Connect 15.4
- Python 3.11.00
The extensions shows everything as green:
The error is still the same:
I am not able to submit the case or at least I am not sure how to do it considering I am using my private Azure Databricks and private e-mail.
Help centre message:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-20-2024 05:58 AM
Got it, I will check with my internal team to validate if there is any issue around.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-20-2024 06:34 AM - edited โ12-20-2024 06:34 AM
Thanks! I appreciate your help
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-20-2024 06:51 AM
I got response from engineering:
grpc_wait_for_shutdown_with_timeout
is actually unrelated. We have an issue open for that, but haven't yet deployed a fix, but this issue should not prevent the execution of sql magic though- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-28-2024 05:28 AM
Hi @Walter_C ,
My setup is as follows:
- Python 3.11.9
- Databricks Runtime: 15.4
- Databricks Connect: 15.4.4
I have modified a cell as follows (python cell, then SQL cell, then python cell):
# Databricks notebook source
df = spark.sql("SELECT 1 AS from__python_cell")
df.show()
# COMMAND ----------
# MAGIC %sql
# MAGIC SELECT 2 AS from_sql_cell
# COMMAND ----------
df = spark.sql("SELECT 3 AS from__python_cell")
df.show()
The output (the error disappeared, but nothing is displayed from SELECT from SQL):
I wonder whether the SQL was executed but simply nothing was displayed or nothing was executed.
If it is executed, do you have an idea how to display the output of the SQL statement in %sql magic command?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-28-2024 08:50 AM
are you trying to see it from the interactive screen as following screenshot:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-28-2024 10:53 AM
Hi @Walter_C ,
Many thanks! I was checking in terminal. After opening Interactive Window I can see the result.
Amazing!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-28-2024 11:04 PM
In reality, there is nothing to do with grpc_wait_for_shutdown_with_timeout. Although we haven't yet implemented a solution, we have an open issue for it, but it shouldn't stop SQL magic from working.
Or is the "Interactive" tab where you encounter this error?
If the local Python version differs from the DBR-required Python version, it might not function.
![](/skins/images/8C2A30E5B696B676846234E4B14F2C7B/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/8C2A30E5B696B676846234E4B14F2C7B/responsive_peak/images/icon_anonymous_message.png)