Magic Commands (%sql) Not Working with Databricks Extension for VS Code

[filipniziol]

Hi Community,

I’ve encountered an issue with the Databricks Extension for VS Code that seems to contradict the documentation. According to the Databricks documentation, the extension supports magic commands like %sql when used with Databricks Connect:

However, in practice, this functionality doesn’t appear to work as expected.

Here’s an example of the code I ran:

# Databricks notebook source

df = spark.sql("SELECT 1 AS from__python_cell")
df.show()

# COMMAND ----------

# MAGIC %sql
# MAGIC SELECT 1 AS from_sql_cell

 The first Python cell executes correctly, displaying the expected result. However, the %sql magic command produces an error:

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR  
E0000 00:00:1734690770.134581    5908 init.cc:229] grpc_wait_for_shutdown_with_timeout() timed out.  

This raises several questions:

1. Is %sql truly supported in the VS Code extension when using Databricks Connect, or is there a misunderstanding in the documentation?
2. Could this be a version compatibility issue between Databricks Connect, the extension, and the runtime?
3. Are there any workarounds to enable the use of %sql in this setup, or should we rely exclusively on spark.sql() in Python cells?

My setup includes:

• Databricks Connect: 13.3.2
• Databricks Runtime: 16.0
• Databricks Extension for VS Code: v2.5.0

[Walter_C]

According to the documentation, the Databricks extension for Visual Studio Code supports the %sql magic command.

However, there are limitations to this support:

• The notebooks magics %r and %scala are not supported and will display an error if called.
• The notebook magic %sql does not support some DML commands, such as Show Tables.

Given your setup (Databricks Connect: 13.3.2, Databricks Runtime: 16.0, Databricks Extension for VS Code: v2.5.0), it appears that the %sql magic command should be supported.

Do you have option to open support case so this can be reviewed further and possibly engage engineering if a bug is detected?

[filipniziol]

Hi @Walter_C ,

Thanks for the answer!

Just to update you I have also adjusted the versions so that all versions match:

• Databricks Runtime 15.4
• Databricks-Connect 15.4
• Python 3.11.00

The extensions shows everything as green:

The error is still the same:

I am not able to submit the case or at least I am not sure how to do it considering I am using my private Azure Databricks and private e-mail.
Help centre message:

[Walter_C]

Got it, I will check with my internal team to validate if there is any issue around.

[filipniziol]

Thanks! I appreciate your help

[Walter_C]

I got response from engineering:

The grpc_wait_for_shutdown_with_timeout is actually unrelated. We have an issue open for that, but haven't yet deployed a fix, but this issue should not prevent the execution of sql magic though

Or do you see this error in the "Interactive" tab?

It's possible that it might not work if the local python version doesn't match DBR required python version