topic Re: I have several thousands of Delta tables in my Production, what is the best way to get counts in Data Engineering

I have several thousands of Delta tables in my Production, what is the best way to get counts

Srikanth_Gupta_ — Tue, 22 Jun 2021 14:56:54 GMT

if I might need a dashboard to see increase in number of rows on day to day basis, also a dashboard that shows size of Parquet/Delta files in my Lake?

Re: I have several thousands of Delta tables in my Production, what is the best way to get counts

sajith_appukutt — Tue, 22 Jun 2021 15:21:32 GMT

The history operation on a delta table returns a collection of operations metrics which includes number of rows and files added/removed between different operations. You could use this in our sql-endpoint workspace and build a dashboard that displays the desired info.

DESCRIBE HISTORY delta.`/data/events/`

Re: I have several thousands of Delta tables in my Production, what is the best way to get counts

brickster_2018 — Tue, 22 Jun 2021 22:53:13 GMT

val db = "database_name"
spark.sessionState.catalog.listTables(db).map(table=>spark.sessionState.catalog.externalCatalog.getTable(table.database.get,table.table)).filter(x=>x.provider.toString().toLowerCase.contains("delta"))

The above code snippet will give the name of all the Delta tables in a database. If the intention is to create a dashboard, then

iterate over all the databases
identify the Delta tables
Call the "DESCRIBE HISTORY" command on each of the delta tables
The details of all the tables and row count can be stored in a Delta table.
Dashboard can query the delta table used to store these details in #4