Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-22-2021 07:35 AM
How did you determine the number of partitions read and the size of these partitions?
The reason I ask is because if your first read the data and then immediately wrote it to another delta table, there is also auto optimize on delta lake, which tries to write 128MB files.
(spark.databricks.delta.autoCompact.maxFileSize)