cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Get number of rows in delta lake table from metadata without count(*)

JDL
New Contributor III

Hello folks,

Is there a way with sql query to get count from delta table metadata without doing count(*) on each of table? Wondering, if this information is stored in any of INFORMATION_SCHEMA tables.

I have a use-case to get counts from 1000's of delta tables and do some further processing based on count.

It doesn't need to exact count but an estimate would be fine too.

4 REPLIES 4

JDL
New Contributor III

Thanks @Retired_mod for your response. I tried the sql query but gives error about the path.

I tried different version of path.
If I specify full path of table, which I retrieve from catalog explorer starting with abfss:// getting below error:

[RequestId=28a3048d-1371-4908-96cd-090961fe9356 ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 'abfss://<>.dfs.core.windows.net/__unitystorage/catalogs/<>/tables/<>' overlaps with managed storage within 'GenerateTemporaryPathCredential' call

If I specify path without abfss:// getting error,
Path must be absolute

Thoughts?

JDL
New Contributor III

Thanks @Retired_mod for detailed information but I am still not clear on resolution.

  1. Table type is shown as MANAGED. Does this mean I can't get count of this table without count(*)?

JDL_0-1701438470267.png

2. I am using path starting with abfss://. Copying the exact path of this table from catalog explorer.

3. I am not familiar if dbutils.fs.ls can be used in SQL Query. I need this information using SQL only due to some limitation.

All IAM permissions are in place.

SSundaram
Contributor

JDL
New Contributor III

Thanks @SSundaram for the link. I need this information via sql query only.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group