- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-24-2022 02:00 PM
I have a path where there is _delta_log and 3 snappy.parquet files. I am trying to read all those .parquet using spark.read.format('delta').load(path) but I am getting data from only one same file all the time. Can't I read from all these files? If so how to achieve this?
- Labels:
-
Databricks SQL
-
Delta Format
-
Delta log
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-25-2022 04:29 AM
the fact there are multiple parquet files does not mean all those files are 'active'. Delta lake can do time travel, meaning you can roll back a delta table to a previous state. To be able to do that, it needs the old data.
That is why old data is not removed, and you can see multiple parquet files which are not used in the most recent version of delta_lake.
you can remove them with the VACUUM command:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-24-2022 05:42 AM
@Werner Stinckens Thanks for the reply and explanation, that was helpful to understand the delta feature.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-25-2022 12:45 PM
Not sure about the best but it helped me to think it differently which I was not aware of.
