autoloader with real time and batch processing concurrently

[sanjay]

Hi,

I have data pipeline which is running continuously, processes the micro batch data and store data in delta lake. This is taking care of any new data.

But at times, I need to process historical data without disturbing real time data processing.

Is there any suggested approach for this scenario. Appreciate any help.

Regards,

Sanjay