โ04-14-2022 09:47 AM
I wanted to setup Autoloader to process files from Azure Data Lake (Blob) automatically whenever new files arrive. For this to work, I wanted to know if AutoLoader requires that the cluster is on all the time.
โ04-19-2022 12:16 AM
Hi @Venkata Ramakrishna Alvakondaโ , If your data only arrives every hour or every day, for example, you can run autoloader in batch mode by specifying the "trigger once" option and then setting up the notebook to run a scheduled job.
In "trigger once" mode, Autoloader still keeps track of new files even when there's not an active cluster running - it just waits to process until you rerun the AutoLoader code manually or as part of a scheduled job.
โ04-14-2022 03:43 PM
@Venkata Ramakrishna Alvakondaโ , No it is not required. Last position is stored in checkpoint file. New files are detected by directory listings or are stored in queue.
โ04-14-2022 07:02 PM
@Hubert Dudekโ , Thank you for your response. My question was: Does the cluster have to be on all the time to take advantage of Auto Loader? What happens if a file arrives in the blob storage and the cluster was down. Does it automatically start the cluster and then invoke the autoloader process to read the file? Or does the next time the cluster starts, it gets picked up?
โ04-19-2022 12:16 AM
Hi @Venkata Ramakrishna Alvakondaโ , If your data only arrives every hour or every day, for example, you can run autoloader in batch mode by specifying the "trigger once" option and then setting up the notebook to run a scheduled job.
In "trigger once" mode, Autoloader still keeps track of new files even when there's not an active cluster running - it just waits to process until you rerun the AutoLoader code manually or as part of a scheduled job.
โ04-19-2022 11:59 AM
Thanks @Kaniz Fatmaโ for the response. Unfortunately I dont have a set frequency for the arrival of files. It is very adhoc. Let me ask you this question. Is it possible for event grid to trigger a Databricks job?
โ04-19-2022 12:31 PM
For sure you can try logic apps for triggering when something is in event grid and then notebook run.
โ11-05-2022 05:40 AM
@Kaniz Fatmaโ , If my cluster is not active, and I have uploaded 50 files in storage location, then where this Auto Loader will list out these 50 files. Will it use any checkpoint location, if yes, then how can I set the checkpoint location in Cloud Storage for these new files identification? Can please tell me the backend process that is used to identifying these new files if my cluster is not active?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group