Lakshay
Databricks Employee
Databricks Employee

As spark processes the data by dividing the data into multiple partitions, so when writing the data no. of part files created will be equal to no. of partitions. If you are doing this outside Autoloader, you can use coalesce to control the no. of partitions but in Autolader, I am not sure if we can use coalesce.

However, you can run optimize command on the delta table to compact the file.