Do we really need Autoloader for batch processing.?

[Mr__D]

Hi All,

It seem AutoLoader is good option for even driven data ingestion but if my job runs only once , do I still need autoloader ? I dont want to spend money to spin a cluster whole day.

I know we have RunOnce option available while running a job but apart from this is there any profit of using Autoloader in batch processing ?

Thanks,

[Lakshay]

Hi @Deepak Bhatt​ , Autoloader is used for Streaming applications. If you have a batch application, you can schedule your job to run as per schedule.

You can check the Autoloader doc here:- https://docs.databricks.com/ingestion/auto-loader/index.html

[Anonymous]

Hi @Deepak Bhatt​ 

Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!

Thanks and regards