cancel
Showing results for 
Search instead for 
Did you mean: 

Do we really need Autoloader for batch processing.?

Mr__D
New Contributor II

Hi All,

It seem AutoLoader is good option for even driven data ingestion but if my job runs only once , do I still need autoloader ? I dont want to spend money to spin a cluster whole day.

I know we have RunOnce option available while running a job but apart from this is there any profit of using Autoloader in batch processing ?

Thanks,

2 REPLIES 2

Lakshay
Honored Contributor III
Honored Contributor III

Hi @Deepak Bhatt​ , Autoloader is used for Streaming applications. If you have a batch application, you can schedule your job to run as per schedule.

You can check the Autoloader doc here:- https://docs.databricks.com/ingestion/auto-loader/index.html

Anonymous
Not applicable

Hi @Deepak Bhatt​ 

Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!

Thanks and regards

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.