cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

how to find the size of a table in python or sql?

ramankr48
Contributor II

let's suppose there is a database db, inside that so many tables are there and , i want to get the size of tables . how to get in either sql, python, pyspark.

even if i have to get one by one it's fine.

1 ACCEPTED SOLUTION

Accepted Solutions

shan_chandra
Databricks Employee
Databricks Employee

@Raman Guptaโ€‹ - could you please try the below

%python
spark.sql("describe detail delta-table-name").select("sizeInBytes").collect()

View solution in original post

6 REPLIES 6

elgeo
Valued Contributor II

DESCRIBE DETAIL table_name returns the sizeInBytes

youssefmrini
Databricks Employee
Databricks Employee

Describe detail give you only the size of the latest snapshot. It's worth running a dbutils.fs.ls

shan_chandra
Databricks Employee
Databricks Employee

@Raman Guptaโ€‹ - Please refer to the below

Calculate the size of the Delta table:

%scala 
import com.databricks.sql.transaction.tahoe._
val deltaLog = DeltaLog.forTable(spark, "dbfs:/delta-table-path")
val snapshot = deltaLog.snapshot                     // the current delta table snapshot
println(s"Total file size (bytes): ${deltaLog.snapshot.sizeInBytes}"

calculate the size of the non delta table:

%scala
spark.read.table("non-delta-table-name").queryExecution.analyzed.stats

Already know in scala.

I wanted this in either python or sql

scala link: https://kb.databricks.com/sql/find-size-of-table.html#:~:text=To%20find%20the%20size%20of%20a%20delt...

shan_chandra
Databricks Employee
Databricks Employee

@Raman Guptaโ€‹ - could you please try the below

%python
spark.sql("describe detail delta-table-name").select("sizeInBytes").collect()

Thanks @Shanmugavel Chandrakasuโ€‹ 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group