Magic Commands (%sql) Not Working with Databricks Extension for VS Code

[filipniziol]

Hi Community,

I’ve encountered an issue with the Databricks Extension for VS Code that seems to contradict the documentation. According to the Databricks documentation, the extension supports magic commands like %sql when used with Databricks Connect:

However, in practice, this functionality doesn’t appear to work as expected.

Here’s an example of the code I ran:

# Databricks notebook source

df = spark.sql("SELECT 1 AS from__python_cell")
df.show()

# COMMAND ----------

# MAGIC %sql
# MAGIC SELECT 1 AS from_sql_cell

 The first Python cell executes correctly, displaying the expected result. However, the %sql magic command produces an error:

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR  
E0000 00:00:1734690770.134581    5908 init.cc:229] grpc_wait_for_shutdown_with_timeout() timed out.  

This raises several questions:

1. Is %sql truly supported in the VS Code extension when using Databricks Connect, or is there a misunderstanding in the documentation?
2. Could this be a version compatibility issue between Databricks Connect, the extension, and the runtime?
3. Are there any workarounds to enable the use of %sql in this setup, or should we rely exclusively on spark.sql() in Python cells?

My setup includes:

• Databricks Connect: 13.3.2
• Databricks Runtime: 16.0
• Databricks Extension for VS Code: v2.5.0

[Walter_C]

According to the documentation, the Databricks extension for Visual Studio Code supports the %sql magic command.

However, there are limitations to this support:

• The notebooks magics %r and %scala are not supported and will display an error if called.
• The notebook magic %sql does not support some DML commands, such as Show Tables.

Given your setup (Databricks Connect: 13.3.2, Databricks Runtime: 16.0, Databricks Extension for VS Code: v2.5.0), it appears that the %sql magic command should be supported.

Do you have option to open support case so this can be reviewed further and possibly engage engineering if a bug is detected?

[filipniziol]

Hi @Walter_C ,

Thanks for the answer!

Just to update you I have also adjusted the versions so that all versions match:

• Databricks Runtime 15.4
• Databricks-Connect 15.4
• Python 3.11.00

The extensions shows everything as green:

The error is still the same:

I am not able to submit the case or at least I am not sure how to do it considering I am using my private Azure Databricks and private e-mail.
Help centre message:

[Walter_C]

Got it, I will check with my internal team to validate if there is any issue around.

[filipniziol]

Thanks! I appreciate your help

[Walter_C]

I got response from engineering:

The grpc_wait_for_shutdown_with_timeout is actually unrelated. We have an issue open for that, but haven't yet deployed a fix, but this issue should not prevent the execution of sql magic though

Or do you see this error in the "Interactive" tab?

It's possible that it might not work if the local python version doesn't match DBR required python version

[filipniziol]

Hi @Walter_C ,

My setup is as follows:

1. Python 3.11.9
2. Databricks Runtime: 15.4
3. Databricks Connect: 15.4.4

I have modified a cell as follows (python cell, then SQL cell, then python cell):

# Databricks notebook source

df = spark.sql("SELECT 1 AS from__python_cell")
df.show()

# COMMAND ----------

# MAGIC %sql
# MAGIC SELECT 2 AS from_sql_cell

# COMMAND ----------

df = spark.sql("SELECT 3 AS from__python_cell")
df.show()

The output (the error disappeared, but nothing is displayed from SELECT from SQL):

I wonder whether the SQL was executed but simply nothing was displayed or nothing was executed.
If it is executed, do you have an idea how to display the output of the SQL statement in %sql magic command?

[Walter_C · Accepted solution]

are you trying to see it from the interactive screen as following screenshot:

[filipniziol]

Hi @Walter_C ,

Many thanks! I was checking in terminal. After opening Interactive Window I can see the result.
Amazing! 

[jack533]

In reality, there is nothing to do with grpc_wait_for_shutdown_with_timeout. Although we haven't yet implemented a solution, we have an open issue for it, but it shouldn't stop SQL magic from working.
Or is the "Interactive" tab where you encounter this error?
If the local Python version differs from the DBR-required Python version, it might not function.