cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databricks All Delta Tables Data Read

Krishna2110
New Contributor II

If we want to read all the data of the databricks tables at single time how can we able to do it.

4 REPLIES 4

szymon_dybczak
Contributor III

Hi @Krishna2110 ,

It's bit unclear to me what is your problem. If you don't use any filter then all data will be read in data frame, as in below.

df = spark.read.delta('my_table')

There is however limitation on number of rows that will be displayed in UI, so maybe you're thinking that not entire data were read?

Or maybe you are asking about situation, where you have set of different tables with the same schema and you would like to query those? Then you can iterate on tables, read each and union results

Krishna2110
New Contributor II

Thankyou for your input.

In the same catalog if there are 40 tables i want to read the data or either schema of all the tables in the same command cell with the help of pyspark.

I have written this code but it was throwing an error 

tables = spark.sql("SHOW TABLES IN ewt_edp_prod.crm_raw")
for table in tables:
    table_name = f"{table.database}.{table.name}"
    try:
        df = spark.table(table_name)
        count = df.count()
        print(f"Table {table_name} is accessible and has {count} rows.")
    except Exception as e:
        print(f"Error accessing table {table_name}: {e}")
 
Can you help me with this code if i have written somewhere wrong.

Yeah, sure. I'll send you code once I'm home

szymon_dybczak
Contributor III

Hi @Krishna2110 ,

Here it is, it should work now

 

tables = spark.sql("SHOW TABLES IN ewt_edp_prod.crm_raw").collect()
for row in tables:
    table_name = f"ewt_edp_prod.{row[0]}.{row[1]}"
    try:
        df = spark.table(table_name)
        count = df.count()
        print(f"Table {table_name} is accessible and has {count} rows.")
    except Exception as e:
        print(f"Error accessing table {table_name}: {e}")

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group