Usability bug in SPARKSQL

vchandm23 — Tue, 13 May 2025 02:23:14 GMT

Hi Databricks & Team,

Spark Cluster: 16.3

Being a databricks partner I am unable to raise a support ticket, hence am positing this here.
pyspark is good in rendering multiple results in a single cell. refer screenshot below

(Screenshot 1)

However, SPARKSQL in the notebook is not working as above: Refer screenshot below

(Screenshot 2)

Expected behaviour: SParkSQL in notebook must (return or display) 2 table results like in pyspark screenshot (1).

Regards,

vchandm23

Re: Usability bug in SPARKSQL

mark_ott — Tue, 11 Nov 2025 01:28:42 GMT

Databricks notebooks currently support multiple outputs per cell in Python (pyspark) but do not provide the same behavior for SQL cells. When running several SQL statements in a single notebook cell, Databricks will only render the output from the last statement. This is a known limitation and has been discussed by both users and Databricks documentation.

Pyspark vs SparkSQL Output Behavior

Python and pyspark cells can show multiple results (e.g., using repeated display() or returning multiple values).
SQL cells, including those using %sql magic, only display the last statement's output. Earlier results are not rendered in the cell output.

Workarounds

To see results for multiple SQL queries, put each query in a separate cell.
For complex workflows, you can execute multiple queries from a single Python cell, capture results with spark.sql(), and use display() on each DataFrame.
There is no built-in option to have native SQL notebook cells output multiple tables in-line, as shown in pyspark, as of Databricks Runtime 16.3.

Expected Behavior

The current behavior—where only the last SQL query’s result is displayed from a notebook SQL cell—is the expected design in Databricks. Multiple result sets in one SQL cell for SparkSQL are not currently supported like they are for pyspark.

If your notebooks require displaying multiple tables at once, the recommended solution is to use Python with spark.sql() for each query or use multiple SQL cells. There is no configuration or workaround to enable multiple SparkSQL table results from a single SQL cell as you hoped.

topic Usability bug in SPARKSQL in Get Started Discussions

Usability bug in SPARKSQL

Re: Usability bug in SPARKSQL

Pyspark vs SparkSQL Output Behavior

Workarounds

Expected Behavior