topic Do we really need Autoloader for batch processing.? in Data Engineering

Do we really need Autoloader for batch processing.?

Mr__D — Fri, 24 Mar 2023 14:41:45 GMT

Hi All,

It seem AutoLoader is good option for even driven data ingestion but if my job runs only once , do I still need autoloader ? I dont want to spend money to spin a cluster whole day.

I know we have RunOnce option available while running a job but apart from this is there any profit of using Autoloader in batch processing ?

Thanks,

Re: Do we really need Autoloader for batch processing.?

Lakshay — Fri, 24 Mar 2023 18:47:26 GMT

Hi @Deepak Bhatt , Autoloader is used for Streaming applications. If you have a batch application, you can schedule your job to run as per schedule.

You can check the Autoloader doc here:- https://docs.databricks.com/ingestion/auto-loader/index.html

Re: Do we really need Autoloader for batch processing.?

Anonymous — Sun, 26 Mar 2023 05:53:45 GMT

Hi @Deepak Bhatt

Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!

Thanks and regards