Databricks All Delta Tables Data Read

[Krishna2110]

If we want to read all the data of the databricks tables at single time how can we able to do it.

[Slash]

Hi @Krishna2110 ,

It's bit unclear to me what is your problem. If you don't use any filter then all data will be read in data frame, as in below.

df = spark.read.delta('my_table')

There is however limitation on number of rows that will be displayed in UI, so maybe you're thinking that not entire data were read?

Or maybe you are asking about situation, where you have set of different tables with the same schema and you would like to query those? Then you can iterate on tables, read each and union results

[Krishna2110]

Thankyou for your input.

In the same catalog if there are 40 tables i want to read the data or either schema of all the tables in the same command cell with the help of pyspark.

I have written this code but it was throwing an error 

tables = spark.sql("SHOW TABLES IN ewt_edp_prod.crm_raw")

for table in tables:

    table_name = f"{table.database}.{table.name}"

    try:

        df = spark.table(table_name)

        count = df.count()

        print(f"Table {table_name} is accessible and has {count} rows.")

    except Exception as e:

        print(f"Error accessing table {table_name}: {e}")

Can you help me with this code if i have written somewhere wrong.

[Slash]

Yeah, sure. I'll send you code once I'm home

[Slash]

Hi @Krishna2110 ,

Here it is, it should work now

tables = spark.sql("SHOW TABLES IN ewt_edp_prod.crm_raw").collect()
for row in tables:
    table_name = f"ewt_edp_prod.{row[0]}.{row[1]}"
    try:
        df = spark.table(table_name)
        count = df.count()
        print(f"Table {table_name} is accessible and has {count} rows.")
    except Exception as e:
        print(f"Error accessing table {table_name}: {e}")