Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2023 11:05 AM
As spark processes the data by dividing the data into multiple partitions, so when writing the data no. of part files created will be equal to no. of partitions. If you are doing this outside Autoloader, you can use coalesce to control the no. of partitions but in Autolader, I am not sure if we can use coalesce.
However, you can run optimize command on the delta table to compact the file.