- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2025 05:00 PM
You’re running into a Databricks SQL results delivery limit—the UI (and even “Download results”) isn’t meant to stream 1.5M × (id, name, 5,000-double array) back to your browser. That’s why SELECT * “works” on Snowflake’s console but not in the DBSQL UI. So don’t measure by returning the whole table.
Here’s how to do a fair, full-scan runtime comparison without changing the data or adding predicates:
Force a full scan but return 1 row
Make the engine read every column and array element and reduce them to a checksum. This avoids UI limits while still measuring scan/compute.
Databricks SQL
-- Optional: avoid cached results
SET use_cached_result = false;
SELECT
SUM(COALESCE(CAST(id AS BIGINT), 0)) AS s_id,
SUM(COALESCE(LENGTH(name), 0)) AS s_name_len,
-- read every element of the 5k-length array<double>
SUM(AGGREGATE(arr, CAST(0.0 AS DOUBLE), (acc, x) -> acc + COALESCE(x, 0.0))) AS s_arr_sum
FROM your_catalog.your_schema.your_table;
- Do the same SQL in snowflake as well to compare
-
This forces a full table scan, column decode, and array traversal on both engines.
-
Measure server-side runtime from each platform’s query history/profile (Databricks: Query Profile / query history; Snowflake: Query History). You’ll also see bytes read / rows scanned to verify it wasn’t a metadata shortcut.