topic Re: Magic Commands (%sql) Not Working with Databricks Extension for VS Code in Data Engineering

Magic Commands (%sql) Not Working with Databricks Extension for VS Code

filipniziol — Fri, 20 Dec 2024 11:06:04 GMT

Hi Community,

I’ve encountered an issue with the Databricks Extension for VS Code that seems to contradict the documentation. According to the Databricks documentation, the extension supports magic commands like %sql when used with Databricks Connect:

However, in practice, this functionality doesn’t appear to work as expected.

Here’s an example of the code I ran:

# Databricks notebook source df = spark.sql("SELECT 1 AS from__python_cell") df.show() # COMMAND ---------- # MAGIC %sql # MAGIC SELECT 1 AS from_sql_cell

The first Python cell executes correctly, displaying the expected result. However, the %sql magic command produces an error:

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR E0000 00:00:1734690770.134581 5908 init.cc:229] grpc_wait_for_shutdown_with_timeout() timed out.

This raises several questions:

Is %sql truly supported in the VS Code extension when using Databricks Connect, or is there a misunderstanding in the documentation?
Could this be a version compatibility issue between Databricks Connect, the extension, and the runtime?
Are there any workarounds to enable the use of %sql in this setup, or should we rely exclusively on spark.sql() in Python cells?

My setup includes:

Databricks Connect: 13.3.2
Databricks Runtime: 16.0
Databricks Extension for VS Code: v2.5.0

Re: Magic Commands (%sql) Not Working with Databricks Extension for VS Code

Walter_C — Fri, 20 Dec 2024 12:03:31 GMT

According to the documentation, the Databricks extension for Visual Studio Code supports the %sql magic command.

However, there are limitations to this support:

The notebooks magics %r and %scala are not supported and will display an error if called.
The notebook magic %sql does not support some DML commands, such as Show Tables.

Given your setup (Databricks Connect: 13.3.2, Databricks Runtime: 16.0, Databricks Extension for VS Code: v2.5.0), it appears that the %sql magic command should be supported.

Do you have option to open support case so this can be reviewed further and possibly engage engineering if a bug is detected?

Re: Magic Commands (%sql) Not Working with Databricks Extension for VS Code

filipniziol — Fri, 20 Dec 2024 12:17:50 GMT

Hi @Walter_C ,

Thanks for the answer!

Just to update you I have also adjusted the versions so that all versions match:

Databricks Runtime 15.4
Databricks-Connect 15.4
Python 3.11.00

The extensions shows everything as green:

The error is still the same:

I am not able to submit the case or at least I am not sure how to do it considering I am using my private Azure Databricks and private e-mail.
Help centre message:

Re: Magic Commands (%sql) Not Working with Databricks Extension for VS Code

Walter_C — Fri, 20 Dec 2024 13:58:22 GMT

Got it, I will check with my internal team to validate if there is any issue around.

Re: Magic Commands (%sql) Not Working with Databricks Extension for VS Code

filipniziol — Fri, 20 Dec 2024 14:34:42 GMT

Thanks! I appreciate your help

Re: Magic Commands (%sql) Not Working with Databricks Extension for VS Code

Walter_C — Fri, 20 Dec 2024 14:51:53 GMT

I got response from engineering:

The grpc_wait_for_shutdown_with_timeout is actually unrelated. We have an issue open for that, but haven't yet deployed a fix, but this issue should not prevent the execution of sql magic though

Or do you see this error in the "Interactive" tab?

It's possible that it might not work if the local python version doesn't match DBR required python version

Re: Magic Commands (%sql) Not Working with Databricks Extension for VS Code

filipniziol — Sat, 28 Dec 2024 13:28:03 GMT

Hi @Walter_C ,

My setup is as follows:

Python 3.11.9
Databricks Runtime: 15.4
Databricks Connect: 15.4.4

I have modified a cell as follows (python cell, then SQL cell, then python cell):

# Databricks notebook source df = spark.sql("SELECT 1 AS from__python_cell") df.show() # COMMAND ---------- # MAGIC %sql # MAGIC SELECT 2 AS from_sql_cell # COMMAND ---------- df = spark.sql("SELECT 3 AS from__python_cell") df.show()

The output (the error disappeared, but nothing is displayed from SELECT from SQL):

I wonder whether the SQL was executed but simply nothing was displayed or nothing was executed.
If it is executed, do you have an idea how to display the output of the SQL statement in %sql magic command?

Re: Magic Commands (%sql) Not Working with Databricks Extension for VS Code

Walter_C — Sat, 28 Dec 2024 16:50:53 GMT

are you trying to see it from the interactive screen as following screenshot:

Re: Magic Commands (%sql) Not Working with Databricks Extension for VS Code

filipniziol — Sat, 28 Dec 2024 18:53:46 GMT

Hi @Walter_C ,

Many thanks! I was checking in terminal. After opening Interactive Window I can see the result.
Amazing!

Re: Magic Commands (%sql) Not Working with Databricks Extension for VS Code

jack533 — Sun, 29 Dec 2024 07:04:47 GMT

In reality, there is nothing to do with grpc_wait_for_shutdown_with_timeout. Although we haven't yet implemented a solution, we have an open issue for it, but it shouldn't stop SQL magic from working.
Or is the "Interactive" tab where you encounter this error?
If the local Python version differs from the DBR-required Python version, it might not function.