topic Re: How to Get the Size of Filtered Rows in Databricks SQL in Data Engineering

How to Get the Size of Filtered Rows in Databricks SQL

vidya_kothavale — Thu, 06 Feb 2025 08:14:50 GMT

I have a query that filters rows from a table based on a timestamp range. The query is as follows:

SELECT COUNT(*) FROM table_name WHERE ts >= '2025-02-04 00:00:00' AND ts < '2025-02-05 00:00:00';

This query returns 10 rows. I need to calculate the total size (in bytes) of these 10 rows.

What is the best approach in Databricks SQL to get the exact size of these filtered rows?
is there a built-in function to get this information?

Re: How to Get the Size of Filtered Rows in Databricks SQL

holly — Thu, 06 Feb 2025 14:37:56 GMT

You'd have to write the results to a table, then do

DESCRIBE DETAIL table_name

and look for the column sizeInBytes

Re: How to Get the Size of Filtered Rows in Databricks SQL

MadhuB — Thu, 06 Feb 2025 17:28:11 GMT

@vidya_kothavale try this code block. Keep in mind to handle the null values.

SELECT SUM(OCTET_LENGTH(CAST(column1 AS STRING)) + OCTET_LENGTH(CAST(column2 AS STRING)) + OCTET_LENGTH(CAST(COALESCE(column3, '0') AS STRING))) as bytes, SUM(OCTET_LENGTH(CAST(column1 AS STRING)) + OCTET_LENGTH(CAST(column2 AS STRING)) + OCTET_LENGTH(CAST(COALESCE(column3, '0') AS STRING))) /POWER(1024,2) as mb, SUM(OCTET_LENGTH(CAST(column1 AS STRING)) + OCTET_LENGTH(CAST(column2 AS STRING)) + OCTET_LENGTH(CAST(COALESCE(column3, '0') AS STRING))) /POWER(1024,3) as gb FROM catalog.schema.table where ts >= '2025-02-04 00:00:00' AND ts < '2025-02-05 00:00:00';

Let me know for anything, else mark it as solution.