Hi @VZLA,

I can indeed confirm there are no overlapping locations. We eventually got a successful run by just increasing the cluster until the micro-batches stayed below 1 hour. I was really thrown off by the error message though, so was wondering if and how it is related to the micro-batch size.

What we are trying to do is process a table's CDF stream and merge changes into another table. In this particular case, we had to reprocess the whole table, which resulted in some micro-batches of over 40 billion records. Looking at the Spark-UI I noticed that it is reading in a 1000 files per micro-batch, so the approach now is to leverage the maxFilesPerTrigger option to tune the micro-batch size.

Thanks