Anonymous
Not applicable
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-08-2023 05:56 PM
@Anant Pingle : Please try using Databricks' Metadata API. This API provides programmatic access to metadata about Databricks objects such as tables, views, and databases.
from pyspark.sql.functions import sum
# Replace "my_catalog" with the name of your catalog
catalog_name = "my_catalog"
# Get a list of all tables in the catalog
tables = spark.catalog.listTables(catalog_name)
# Compute the size of each table and sum them up
total_size = sum([spark.table(table.database + "." + table.name).count() for table in tables])
print(f"The total size of {catalog_name} is {total_size} rows.")Link to the API documentation: https://docs.databricks.com/dev-tools/api/latest/workspace.html