mmayorga
Databricks Employee
Databricks Employee

hi @Michał 

with Autoloader is designed to provide that exactly-once ingestion and can resume processing from the last successful checkpoint after a failure happens without starting from the beginning. So when you restart the stream it continues where it left off (unless you run a full refresh); just make sure to define your "checkpointLocation" option.

As far as I understand, there is no support for row-level restarts, as an alternative you can leverage the option "modifiedAfter". More details can be found in the Recovery Options in the documentation.

I hope this helps! 

Thank you!

View solution in original post