topic Re: Autoloader cluster in Data Engineering

Autoloader cluster

RK_AV — Thu, 14 Apr 2022 16:47:36 GMT

I wanted to setup Autoloader to process files from Azure Data Lake (Blob) automatically whenever new files arrive. For this to work, I wanted to know if AutoLoader requires that the cluster is on all the time.

Re: Autoloader cluster

Hubert-Dudek — Thu, 14 Apr 2022 22:43:47 GMT

@Venkata Ramakrishna Alvakonda , No it is not required. Last position is stored in checkpoint file. New files are detected by directory listings or are stored in queue.

Re: Autoloader cluster

RK_AV — Fri, 15 Apr 2022 02:02:37 GMT

@Hubert Dudek , Thank you for your response. My question was: Does the cluster have to be on all the time to take advantage of Auto Loader? What happens if a file arrives in the blob storage and the cluster was down. Does it automatically start the cluster and then invoke the autoloader process to read the file? Or does the next time the cluster starts, it gets picked up?

Re: Autoloader cluster

RK_AV — Tue, 19 Apr 2022 18:59:27 GMT

Thanks @Kaniz Fatma for the response. Unfortunately I dont have a set frequency for the arrival of files. It is very adhoc. Let me ask you this question. Is it possible for event grid to trigger a Databricks job?

Re: Autoloader cluster

Hubert-Dudek — Tue, 19 Apr 2022 19:31:00 GMT

For sure you can try logic apps for triggering when something is in event grid and then notebook run.

Re: Autoloader cluster

asif5494 — Sat, 05 Nov 2022 12:40:17 GMT

@Kaniz Fatma , If my cluster is not active, and I have uploaded 50 files in storage location, then where this Auto Loader will list out these 50 files. Will it use any checkpoint location, if yes, then how can I set the checkpoint location in Cloud Storage for these new files identification? Can please tell me the backend process that is used to identifying these new files if my cluster is not active?