Hi @szymon_dybczak ,

Thanks for your reply.

I did try multiple date formats supported in Azure Databricks. I also tried using 
.option('cloudFiles.includeExistingFiles', False). It still didn't work.

I cleared the checkpoints each time.