cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

how to find the size of a table in python or sql?

ramankr48
Contributor II

let's suppose there is a database db, inside that so many tables are there and , i want to get the size of tables . how to get in either sql, python, pyspark.

even if i have to get one by one it's fine.

1 ACCEPTED SOLUTION

Accepted Solutions

shan_chandra
Esteemed Contributor
Esteemed Contributor

@Raman Gupta​ - could you please try the below

%python
spark.sql("describe detail delta-table-name").select("sizeInBytes").collect()

View solution in original post

6 REPLIES 6

elgeo
Valued Contributor II

DESCRIBE DETAIL table_name returns the sizeInBytes

youssefmrini
Honored Contributor III
Honored Contributor III

Describe detail give you only the size of the latest snapshot. It's worth running a dbutils.fs.ls

shan_chandra
Esteemed Contributor
Esteemed Contributor

@Raman Gupta​ - Please refer to the below

Calculate the size of the Delta table:

%scala 
import com.databricks.sql.transaction.tahoe._
val deltaLog = DeltaLog.forTable(spark, "dbfs:/delta-table-path")
val snapshot = deltaLog.snapshot                     // the current delta table snapshot
println(s"Total file size (bytes): ${deltaLog.snapshot.sizeInBytes}"

calculate the size of the non delta table:

%scala
spark.read.table("non-delta-table-name").queryExecution.analyzed.stats

Already know in scala.

I wanted this in either python or sql

scala link: https://kb.databricks.com/sql/find-size-of-table.html#:~:text=To%20find%20the%20size%20of%20a%20delt...

shan_chandra
Esteemed Contributor
Esteemed Contributor

@Raman Gupta​ - could you please try the below

%python
spark.sql("describe detail delta-table-name").select("sizeInBytes").collect()

Thanks @Shanmugavel Chandrakasu​ 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!