topic Re: Reading multiple parquet files from same _delta_log under a path in Data Engineering

Reading multiple parquet files from same _delta_log under a path

KKo — Mon, 24 Jan 2022 22:00:16 GMT

I have a path where there is _delta_log and 3 snappy.parquet files. I am trying to read all those .parquet using spark.read.format('delta').load(path) but I am getting data from only one same file all the time. Can't I read from all these files? If so how to achieve this?

Re: Reading multiple parquet files from same _delta_log under a path

-werners- — Tue, 25 Jan 2022 12:29:02 GMT

the fact there are multiple parquet files does not mean all those files are 'active'. Delta lake can do time travel, meaning you can roll back a delta table to a previous state. To be able to do that, it needs the old data.

That is why old data is not removed, and you can see multiple parquet files which are not used in the most recent version of delta_lake.

you can remove them with the VACUUM command:

https://docs.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/language-manual/delta-vacuum

Re: Reading multiple parquet files from same _delta_log under a path

KKo — Thu, 24 Feb 2022 13:42:16 GMT

@Werner Stinckens Thanks for the reply and explanation, that was helpful to understand the delta feature.

Re: Reading multiple parquet files from same _delta_log under a path

KKo — Fri, 25 Feb 2022 20:45:24 GMT

Not sure about the best but it helped me to think it differently which I was not aware of.