topic How to reset an autoloader? in Data Engineering

How to reset an autoloader?

Scouty — Thu, 10 Feb 2022 10:21:16 GMT

i'm using an autoloader with Azure Databricks:

df = (spark.readStream.format("cloudFiles")

.options(**cloudfile)

.load("abfss://dev@std******.dfs.core.windows.net/**/*****))

at my target checkpointLocation folder there are some files and subdirs created as a result.

It will detect and process new files which is OK.

Also when I restart my cluster it will again process only the new files, which is OK.

But if I want to restart the autoloader in order to re-process all files from the source folder again I could not find anything how to do so.

Can someone please give me a hint.

Re: How to reset an autoloader?

AmanSehgal — Thu, 10 Feb 2022 12:58:00 GMT

Change the checkpoint location or delete the existing checkpoint location.

The new checkpoint location implies that previous stream has been abandoned and new stream has been started.

Re: How to reset an autoloader?

Anonymous — Thu, 10 Feb 2022 14:55:58 GMT

@Aman Sehgal - My name is Piper, and I'm one of the moderators for Databricks. I wanted to jump in real quick to thank you for being so generous with your knowledge. 🙂