Reading multiple parquet files from same _delta_log under a path

[KKo]

I have a path where there is _delta_log and 3 snappy.parquet files. I am trying to read all those .parquet using spark.read.format('delta').load(path) but I am getting data from only one same file all the time. Can't I read from all these files? If so how to achieve this?

[-werners- · Accepted solution]

the fact there are multiple parquet files does not mean all those files are 'active'. Delta lake can do time travel, meaning you can roll back a delta table to a previous state. To be able to do that, it needs the old data.

That is why old data is not removed, and you can see multiple parquet files which are not used in the most recent version of delta_lake.

you can remove them with the VACUUM command:

https://docs.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/language-manual/delta-vacuu...

[KKo]

@Werner Stinckens​ Thanks for the reply and explanation, that was helpful to understand the delta feature.

[KKo]

Not sure about the best but it helped me to think it differently which I was not aware of.