Tharun-Kumar
Databricks Employee
Databricks Employee

Hi @Gilg 

You mentioned that micro-batch time is around 12 minutes recently. Do we also see jobs/stages with 12 minutes in the spark ui. If that is the case, then the processing of the file itself takes 12 minutes. If not, the 12 minutes is spent on listing the directory and maintaining the checkpoint.