topic Re: When would you not want to use autoloader? in Data Engineering

When would you not want to use autoloader?

StephanieAlba — Tue, 17 Oct 2023 13:49:35 GMT

I am genuinely curious why would you ever not use Autoloader? I see it in one-off downloads of course. When you pull data from another platform, say Salesforce, is it better to append to a table without Autoloader?

There must be cases I am missing. Thoughts??

Re: When would you not want to use autoloader?

Tharun-Kumar — Wed, 18 Oct 2023 04:42:10 GMT

@StephanieAlba

Auto Loader is primarily used for processing files automatically as they arrive in cloud storage.

You can refer to this document on AutoLoader - https://docs.databricks.com/en/ingestion/auto-loader/index.html

Re: When would you not want to use autoloader?

-werners- — Wed, 18 Oct 2023 05:32:18 GMT

Autoloader is pretty handy, but not open source. That is one reason f.e.
Another reason is f.e. if you cannot guarantee lexicographically generated files, or you do not want to use streaming, or you do not land your raw data into a data lake (read from db directly f.e.).
I still use a lot of batch processing using /.../YYYY/MM/DD directory structure without autoloader.
Not because I don´t like it (I do use it for new setups), but because at the time it just did not exist and for a 1x/day batch job autoloader is not necessary.