It seem AutoLoader is good option for even driven data ingestion but if my job runs only once , do I still need autoloader ? I dont want to spend money to spin a cluster whole day.
I know we have RunOnce option available while running a job but apart from this is there any profit of using Autoloader in batch processing ?
Hi @Deepak Bhatt , Autoloader is used for Streaming applications. If you have a batch application, you can schedule your job to run as per schedule.
You can check the Autoloader doc here:- https://docs.databricks.com/ingestion/auto-loader/index.html
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!