Tharun-Kumar
Databricks Employee
Databricks Employee

@bzh 

I highly suspect that a single file has 3 million records. To confirm if the data is from a single file or multiple files, could you add a new column with the value as input_file_name(). This would help us to understand whether the config maxFilesPerTrigger is being considered or not.

Document - https://docs.databricks.com/en/sql/language-manual/functions/input_file_name.html