I'm using this code to read data from an ADLS Gen2 location. There are txt files present in sub-folders in the container. df_stream = spark.readStream \
.format("cloudFiles") \
.option("cloudFiles.format", "text") \
.optio...
Hi @ManojkMohan ,Thanks for your reply.The checkpoint location is set to a new path. I tried multiple approaches and cleared the checkpoint location each time so it's considered as the first run. I followed all the steps you've mentioned above. Stil...
Hi @szymon_dybczak ,Thanks for your reply.I did try multiple date formats supported in Azure Databricks. I also tried using .option('cloudFiles.includeExistingFiles', False). It still didn't work.I cleared the checkpoints each time.
