Showing results for 
Search instead for 
Did you mean: 

Do we really need Autoloader for batch processing.?

New Contributor II

Hi All,

It seem AutoLoader is good option for even driven data ingestion but if my job runs only once , do I still need autoloader ? I dont want to spend money to spin a cluster whole day.

I know we have RunOnce option available while running a job but apart from this is there any profit of using Autoloader in batch processing ?



Honored Contributor III
Honored Contributor III

Hi @Deepak Bhatt​ , Autoloader is used for Streaming applications. If you have a batch application, you can schedule your job to run as per schedule.

You can check the Autoloader doc here:-

Not applicable

Hi @Deepak Bhatt​ 

Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!

Thanks and regards

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.