RKNutalapati
Valued Contributor

Hi @Aman Sehgal​  : I am not sure if you have already explored databricks Autoloader and the limitation for your use case. otherwise you can try using Autoloader and avoid multiple processes.