Krishna_S
Databricks Employee
Databricks Employee

 

You’re running into a Databricks SQL results delivery limit—the UI (and even “Download results”) isn’t meant to stream 1.5M × (id, name, 5,000-double array) back to your browser. That’s why SELECT * “works” on Snowflake’s console but not in the DBSQL UI. So don’t measure by returning the whole table.

Here’s how to do a fair, full-scan runtime comparison without changing the data or adding predicates:

Force a full scan but return 1 row

Make the engine read every column and array element and reduce them to a checksum. This avoids UI limits while still measuring scan/compute.

Databricks SQL 

-- Optional: avoid cached results
SET use_cached_result = false;

SELECT
  SUM(COALESCE(CAST(id AS BIGINT), 0))                                        AS s_id,
  SUM(COALESCE(LENGTH(name), 0))                                              AS s_name_len,
  -- read every element of the 5k-length array<double>
  SUM(AGGREGATE(arr, CAST(0.0 AS DOUBLE), (acc, x) -> acc + COALESCE(x, 0.0))) AS s_arr_sum
FROM your_catalog.your_schema.your_table;
  • Do the same SQL in snowflake as well to compare
  • This forces a full table scan, column decode, and array traversal on both engines.

  • Measure server-side runtime from each platform’s query history/profile (Databricks: Query Profile / query history; Snowflake: Query History). You’ll also see bytes read / rows scanned to verify it wasn’t a metadata shortcut.

View solution in original post