Usability bug in SPARKSQL

[vchandm23]

Hi Databricks & Team, 

Spark Cluster: 16.3

Being a databricks partner I am unable to raise a support ticket, hence am positing this here. 
pyspark is good in rendering multiple results in a single cell. refer screenshot below 

(Screenshot 1)

However, SPARKSQL in the notebook is not working as above: Refer screenshot below

(Screenshot 2)

Expected behaviour: SParkSQL in notebook must (return or display) 2 table results like in pyspark screenshot (1).

Regards,

vchandm23

[mark_ott]

Databricks notebooks currently support multiple outputs per cell in Python (pyspark) but do not provide the same behavior for SQL cells. When running several SQL statements in a single notebook cell, Databricks will only render the output from the last statement. This is a known limitation and has been discussed by both users and Databricks documentation.​

Pyspark vs SparkSQL Output Behavior

• Python and pyspark cells can show multiple results (e.g., using repeated display() or returning multiple values).

• SQL cells, including those using %sql magic, only display the last statement's output. Earlier results are not rendered in the cell output.

Workarounds

• To see results for multiple SQL queries, put each query in a separate cell.

• For complex workflows, you can execute multiple queries from a single Python cell, capture results with spark.sql(), and use display() on each DataFrame.​

• There is no built-in option to have native SQL notebook cells output multiple tables in-line, as shown in pyspark, as of Databricks Runtime 16.3.​

Expected Behavior

• The current behavior—where only the last SQL query’s result is displayed from a notebook SQL cell—is the expected design in Databricks. Multiple result sets in one SQL cell for SparkSQL are not currently supported like they are for pyspark.​

If your notebooks require displaying multiple tables at once, the recommended solution is to use Python with spark.sql() for each query or use multiple SQL cells. There is no configuration or workaround to enable multiple SparkSQL table results from a single SQL cell as you hoped.