Reading multiple parquet files from same _delta_log under a path

[KKo]

I have a path where there is _delta_log and 3 snappy.parquet files. I am trying to read all those .parquet using spark.read.format('delta').load(path) but I am getting data from only one same file all the time. Can't I read from all these files? If so how to achieve this?

[Kaniz_Fatma]

Hi @Kris Koirala​ ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[-werners- · Accepted solution]

the fact there are multiple parquet files does not mean all those files are 'active'. Delta lake can do time travel, meaning you can roll back a delta table to a previous state. To be able to do that, it needs the old data.

That is why old data is not removed, and you can see multiple parquet files which are not used in the most recent version of delta_lake.

you can remove them with the VACUUM command:

https://docs.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/language-manual/delta-vacuu...

[KKo]

@Werner Stinckens​ Thanks for the reply and explanation, that was helpful to understand the delta feature.

[Kaniz_Fatma]

Hi @Kris Koirala​ , It seems your problem is resolved. Can we mark @Werner Stinckens​ 's explanation as the best?

[KKo]

Not sure about the best but it helped me to think it differently which I was not aware of.