In delta while query on delta unable to see previous partition where as while reading data using parquet file format it is showing whole partition data column .

[Gaurav_784295]

In delta while query on delta unable to see previous partition where as while reading data using parquet file format it is showing whole partition data column .

Delta Format = spark.read.format("delta").load("") 

Parquet Format ==> spark.read.parquet("")

[Anonymous]

@Gaurav Rawat​ :

Delta tables are designed to support ACID transactions and provide a versioned, transactional view of data. When you query a Delta table, you see the version of the data that was current at the time the query started. If other transactions have modified the table since then, your query will not see those changes. However, if you re-run the query at a later time, you will see the changes that were made up to that point.

Parquet files, on the other hand, are a columnar file format used for storing data. When you read a Parquet file, you are reading the entire contents of the file, including all partitions. If you have multiple Parquet files, each containing a partition of the data, you can read them all and combine them to see the entire dataset.

So, if you are seeing different results when querying a Delta table versus reading Parquet files, it may be because you are querying a different version of the data. It is also possible that the Parquet files contain different data than the Delta table due to changes made after the Delta table was created.

[Anonymous]

Hi @Gaurav Rawat​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!