topic Re: Configure Autoloader with the file notification mode for production in Data Engineering

Configure Autoloader with the file notification mode for production

Chris_Konsur — Tue, 18 Oct 2022 14:19:04 GMT

I configured ADLS Gen2 standard storage and successfully configured Autoloader with the file notification mode.

In this document

https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html

"ADLS Gen2 provides different event notifications for files appearing in your Gen2 container.

Auto Loader listens for the FlushWithClose event for processing a file.

Do I need to do anything with this FlushWithClose event or Autoloader

when configured in the file notification mode=True, automatically listen to the FlushWithClose event REST API?

Next, in the same document,Databricks recommends triggering regular backfills with Auto Loader by using the cloudFiles.backfillInterval option guarantees that all files are discovered within a given SLA if data completeness is required. Triggering regular backfills does not cause duplicates.

From <https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html>

Finally, I found this article how to use Auto Loader Resources Manager Scala API

https://www.mssqltips.com/sqlservertip/6965/databricks-auto-loader-manager/

Do you know if this Resource Mgr is available in Python?

Re: Configure Autoloader with the file notification mode for production

Ryan_Chynoweth — Mon, 05 Dec 2022 18:46:34 GMT

Hi, @Chris Konsur.

You do not need anything with the FlushWithClose event REST API that is just the event type that we listen to.

As for backfill setting, this is for handling late data or late event that are being triggered. This setting largely depends on your SLAs. The setting determines how often you should be doing a full reconciliation of the data that has been processed. I would also recommend checking our the incremental file listing as well.

As for the resource manager, I do not believe there is a Python version.

Re: Configure Autoloader with the file notification mode for production

Chris_Konsur — Mon, 05 Dec 2022 22:36:58 GMT

Excellent, thank you, Ryan!